Papers
Topics
Authors
Recent
Search
2000 character limit reached

Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

Published 10 Sep 2026 in cs.CV and cs.LG | (2609.11638v1)

Abstract: We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing. Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p video generation, generation with dynamic references that can be updated at any moment, and stronger instruction following, such as dancing. Vidu S2-Editing supports editing a video stream in real time, including style rendering, clothing replacement, character replacement, and background replacement. Experiments show that Vidu S2 outperforms all baselines. A playable online demo is available at https://vidu.com/vidu-stream.

Summary

  • The paper introduces Vidu S2, a system for real-time, interactive video generation and editing that maintains high perceptual quality and temporal stability using diffusion-based models.
  • The system achieves state-of-the-art results in generating audio-driven digital characters and editing live streams with tasks like style transfer, background replacement, and virtual try-on.
  • Both Vidu S2-Avatar and Vidu S2-Editing components integrate advanced techniques, including Self-Replay Forcing and VLM-mediated interaction, to enhance real-time performance and interactive capabilities.

Vidu S2 presents a unified system for real-time interactive video generation and editing, comprising Vidu S2-Avatar and Vidu S2-Editing. The paper’s central claim is that diffusion-based video models can move beyond offline, one-shot synthesis toward continuous, user-driven audiovisual interaction while retaining high perceptual quality and long-horizon temporal stability. Vidu S2-Avatar generates audio-driven digital characters at 720p and supports reference updates during an ongoing stream; Vidu S2-Editing transforms incoming video in real time through style transfer, virtual try-on, subject replacement, and background replacement. The system also provides an exploratory pipeline for converting generated or edited monocular streams into synchronized spatial video (2609.11638).

System scope and design objectives

The paper identifies the train–test mismatch in streaming diffusion generation as the principal technical obstacle. Conventional diffusion video models denoise complete sequences offline, whereas an interactive system must generate successive segments conditioned on its own imperfect historical outputs. Errors can therefore accumulate across segments, producing identity drift, background instability, motion discontinuity, or eventual collapse. Vidu S2 addresses this issue through causal adaptation, self-generated trajectory training, preference optimization, and an inference stack specifically engineered for low latency.

The two model variants share an emphasis on segment-level conditioning and causal temporal processing, but their conditioning structures differ. Vidu S2-Avatar jointly models video and audio, conditioned on a persistent reference image and temporally ordered textual or speech-derived instructions. Vidu S2-Editing conditions the target video on the temporally aligned source stream, an optional reference image, and an editing instruction. The distinction is important: Avatar must synthesize motion, appearance, and audio, whereas Editing must preserve the source stream’s timing and motion while modifying selected semantic attributes.

The paper’s system-level architecture combines model design with serving infrastructure rather than treating real-time performance as a consequence of architectural simplification alone.

Figure 1

Figure 1: Overview of Vidu S2, including real-time avatar generation, streaming video editing, and spatial-video conversion.

Data construction and temporal supervision

The data pipeline is a major component of the contribution. For Vidu S2-Avatar, the authors retain the five-stage processing structure inherited from Vidu S1—clipping, filtering, speech processing, captioning, and embedding—but substantially modify the data-selection and annotation procedures. The training corpus is expanded with solo dance videos and 2D/3D animation to improve full-body motion and expressive behavior. Videos with smooth camera motion are retained through background stabilization rather than discarded, increasing motion diversity without directly exposing the model to uncontrolled background drift.

A particularly consequential design choice is the replacement of structured scene descriptions with temporally ordered dense captions. Captions specify actions, temporal boundaries, and outcomes in chronological order. Chunk boundaries are therefore determined by event transitions rather than fixed-duration or speech-aligned intervals. This representation is intended to improve causal instruction following: the model receives a description of what occurs in the current segment and can preserve the resulting state in later segments. The paper reports that candidate cuts can be assessed with a second-stage VLM procedure while keeping the false rejection rate below 2%, although it does not provide a complete ablation isolating the contribution of this procedure to final generation quality.

The data pipeline also evaluates actual visual clarity rather than relying on nominal resolution. Resolution, frame rate, codec, pixel format, bit depth, bitrate, texture detail, edge sharpness, and compression artifacts are combined into a quality score. This is directly relevant to 720p streaming, where upscaling nominally high-resolution but heavily compressed material would not provide suitable supervision.

For Vidu S2-Editing, 800,000 filtered videos are divided into four disjoint subsets of 200,000 examples for style transfer, subject replacement, background replacement, and virtual try-on. The authors construct synthetic pairs using multiple editing models and retain examples through post-filtering and comparative selection. Style-transfer data are generated using surface-normal videos and reference images, allowing the synthetic editor to preserve source geometry and motion while changing appearance. The resulting procedure reportedly generalizes across more than 50 photorealistic and non-photorealistic styles.

Figure 2

Figure 2: Data preparation pipelines for Avatar and Editing, including temporal clipping, clarity filtering, background stabilization, captioning, and synthetic editing-pair construction.

Vidu S2-Avatar

Vidu S2-Avatar is an audio-visual joint Diffusion Transformer for reference-conditioned streaming generation. A reference image is shared across segments to maintain identity and appearance, while each segment receives its own textual condition. The model is trained bidirectionally for quality and then adapted to block-wise causal attention for streaming inference. During causal generation, a segment attends to its reference, current condition, and valid historical states.

The transition from bidirectional training to causal inference uses a hybrid of Teacher Forcing and Diffusion Forcing. Teacher Forcing supplies clean historical states, while Diffusion Forcing injects noise into historical states at sampled noise levels. This exposes the model to a broader distribution of contexts and partially addresses the mismatch between clean training histories and noisy or imperfect inference histories.

Self-Replay Forcing

The main methodological contribution is Self-Replay Forcing (SRF). The model first performs a long autoregressive rollout using its own inference procedure. The rollout and its KV caches are detached, preventing the memory and computational cost of backpropagating through the entire trajectory. The generated segments are then independently re-noised and replayed through a gradient-enabled causal pass. Within this replay pass, the segment representations remain connected, allowing gradients from later segments to affect preceding replayed segments.

This differs from simply training on detached self-generated histories. Detached histories reproduce the inference distribution but do not permit cross-segment gradient flow. SRF attempts to combine both properties: on-policy historical contexts and differentiable temporal credit assignment within the replayed trajectory. DMD supervision is applied to the replayed blocks, supplemented by a perceptual loss intended to prevent mode collapse and preserve diversity.

The paper presents SRF as a response to two limitations attributed to prior self-forcing approaches: clean rather than noised self-generated histories, and the absence of gradient propagation through the replayed sequence. However, the reported experiments do not include a detailed component ablation for SRF, Diffusion Forcing, perceptual regularization, and preference optimization separately. Consequently, the aggregate improvements cannot be attributed quantitatively to SRF alone.

Preference optimization and resolution refinement

Preference optimization is applied at both stages of training. Diffusion DPO improves the bidirectional teacher with respect to visual fidelity, expression, motion naturalness, and audio–visual synchronization. Streaming Negative-aware Fine-Tuning then operates on self-generated trajectories to align the causal model with inference-time states. This two-stage arrangement reflects the paper’s broader strategy: first construct a high-quality teacher, then preserve its quality under causal streaming constraints.

The causal backbone operates at lower resolution, while a one-step latent-space Refiner produces 720p output. The Refiner uses asymmetric cache noise levels: a high-noise cache in the backbone propagates coarse temporal structure, while a lower-noise high-resolution cache preserves local appearance and identity. This separates long-range motion propagation from spatial detail restoration. The authors report real-time 720p generation at 25–42 FPS, improving on Vidu S1’s 540p output. The paper does not, however, specify a single standardized end-to-end latency figure across hardware configurations, which complicates direct comparison of responsiveness.

Dynamic reference interaction

Vidu S2-Avatar supports reference images introduced or changed during a stream. References may specify handheld objects, clothing, accessories, backgrounds, or a new scene. A VLM agent interprets the user’s instruction and reference image, generates a structured prompt, observes subsequent frames, and revises later prompts according to whether an action has completed.

This agentic layer maintains persistent state explicitly. For example, after a character picks up a cup, subsequent prompts preserve the fact that the cup remains held unless the user requests its removal. Similar state tracking is used for taking off and putting on accessories, changing clothing, and moving between scenes. The mechanism is therefore not merely prompt expansion; it is a VLM-mediated state-estimation and prompt-replanning loop.

Figure 3

Figure 3: VLM-mediated agentic control loop for prompt generation, visual verification, and state-preserving interaction.

Figure 4

Figure 4: Dynamic reference and accessory control, including object replacement, scene transitions, and removing and restoring a hat.

Vidu S2-Editing

Vidu S2-Editing performs instruction- and reference-conditioned editing on a live source stream. Its defining architectural constraint is frame-aligned attention: each target frame reads only the source frame at the same temporal position, while the reference image remains accessible to all target frames. This preserves the source motion and timing while allowing the appearance specified by the reference to remain temporally consistent.

The model is first trained bidirectionally. Target tokens interact bidirectionally within the target sequence, but source-target interaction is restricted to aligned frame pairs. The model is subsequently adapted to causal streaming using the same hybrid forcing and SRF framework as Vidu S2-Avatar. During inference, each source frame is processed with its target frame and is not retained indefinitely in the cache, which constrains memory growth.

The editing formulation covers four tasks:

  • Style transfer: global appearance changes while retaining source geometry and motion.
  • Virtual try-on: clothing replacement under articulated motion and occlusion.
  • Subject replacement: replacement of a person or character while preserving pose and interaction.
  • Background replacement: scene substitution while retaining foreground motion and camera behavior.

The serving system uses inter-module scheduling to share GPU capacity among the VAE encoder, backbone, Refiner, and decoder. Because these modules are active at different times, dynamic allocation reduces idle capacity relative to static module-to-GPU placement.

Spatial video generation and editing

Vidu S2 extends both models to spatial video by producing synchronized left- and right-eye views. For Avatar, a monocular stream is generated first. Per-frame depth is estimated, horizontal disparity is applied in opposite directions to generate the two views, and lightweight hole filling and temporal depth stabilization reduce artifacts near disocclusions.

For Editing, the system supports both monocular and stereoscopic inputs. Monocular video is edited and then converted to stereo. For stereoscopic input, the two views are concatenated, edited jointly under shared references, and split again into synchronized left- and right-eye outputs. This preserves a common editing decision across views, although the paper does not report quantitative stereo-consistency metrics, disparity error, or user studies in head-mounted displays.

The authors explicitly acknowledge that spatial video imposes stricter requirements than monocular streaming. Large fields of view require higher resolution, and passthrough applications are sensitive to end-to-end latency because head motion can become desynchronized from the displayed view. The current system is therefore best interpreted as a feasibility demonstration rather than a complete solution to low-latency, high-resolution immersive video.

Figure 5

Figure 5: Representative spatial-video results showing synchronized generated left- and right-eye views.

Experimental results

The evaluation combines public benchmarks, internal long-horizon tests, and randomized human preference comparisons. For streaming digital characters, the public StreamAV-Bench evaluation reports visual, audio, cross-modal, instruction-following, and consistency metrics. For editing, the evaluation uses OpenVE-Bench, Sparkle-Bench, RefVIE-Bench, and the ViViD virtual try-on test set.

On StreamAV-Bench, Vidu S2-Avatar achieves the best reported value in every metric. Its scores are 0.687 for Visual Aesthetics, 3.370 for Visual Quality, 7.138 for Production Quality, 3.286 for Audio Quality, 0.353 for Audio–Visual Alignment, 0.617 for Audio–Visual Synchronization error, 2.985 for Audio Instruction Fulfillment, 0.998 for Subject Consistency, and 0.993 for Background Consistency. The strongest baseline, Live Avatar, reaches 0.661, 3.295, 7.133, 3.079, 0.116, 1.145, 2.745, 0.997, and 0.989 on the corresponding metrics. These results indicate that the advantage is not restricted to appearance: the model also improves audio quality, synchronization, instruction fulfillment, and long-horizon stability. The AVAlign metric is embedding-based and the Gemini-derived metrics are model-based, so the results should not be interpreted as a complete substitute for human evaluation.

For video editing, Vidu S2-Editing obtains the highest score on every reported Sparkle-Bench dimension. Its Overall score is 3.74, compared with 3.67 for Decart-Lucy2.5 and 3.57 for Kiwi-Edit 5B. It reaches 4.00 for global instruction, 3.34 for visual quality, 3.98 for foreground instruction, 4.00 for foreground motion, 3.37 for background dynamics, and 3.76 for background visual quality.

On the joint OpenVE–RefVIE evaluation, Vidu S2-Editing achieves a Joint Overall score of 4.26, exceeding Bernini-R 14B’s 3.92 by 0.34. It obtains 4.71 for global style, 4.14 for background change, 4.42 for OpenVE Overall, and 3.78 for RefVIE Overall. The reported RefVIE score exceeds Decart-Lucy2.5 by 0.07. On the ViViD unpaired virtual try-on test set, Vidu S2-Editing records a VFID of 9.9515, compared with 19.5131 for CatVTON and 21.8032 for ViViD. Because VFID is a distributional metric, this result supports improved aggregate video similarity but does not independently establish garment fidelity under every motion or occlusion pattern.

The internal human evaluations reinforce the public-benchmark results. In digital-character comparisons, Vidu S2-Avatar is preferred in overall quality in 85.7% of paired judgments against Runway Character GWM-1 and in 100% of comparisons against both PixVerse Image Avatar and HeyGen. Preference rates for semantic adherence range from 71.4% to 100%, while video-quality preference ranges from 57.1% to 71.4%. For audio–visual synchronization, Vidu S2-Avatar is preferred or judged equivalent in all reported comparisons against PixVerse and HeyGen.

The duration-stratified evaluation reports the highest mean rating for Vidu S2-Avatar across overall quality, consistency, video quality, motion quality, and emotional expression from 10 to 90 seconds. This is an important result because it tests degradation with duration rather than only short-clip quality. Nevertheless, the evaluation does not establish indefinite stability, and the paper’s own longer-horizon tests are described primarily through internal benchmarks and qualitative examples.

Figure 6

Figure 6

Figure 6

Figure 6

Figure 6

Figure 6

Figure 6: Duration-stratified human ratings showing sustained Avatar quality across streams extending to 90 seconds.

Figure 7

Figure 7

Figure 7: GSB human-preference comparisons between Vidu S2-Avatar and commercial digital-character systems.

For editing, the internal benchmark contains 150 paired cases. Vidu S2-Editing achieves a mean consistency score of 3.56, compared with 2.59 for Decart-Lucy2.5 and 1.67 for XMax-X2.0. It receives the larger preference share across overall quality, video quality, temporal consistency, and semantic adherence.

Figure 8

Figure 8

Figure 8: GSB comparisons between Vidu S2-Editing and commercial streaming editors across quality, consistency, and semantic criteria.

Qualitative behavior

The qualitative cases are consistent with the quantitative emphasis on identity preservation and temporal coherence. Avatar examples show retention of facial identity, hairstyle, eye color, attire, facial proportions, hand geometry, and small-object interactions. The authors report that competing systems exhibit facial drift, body-proportion changes, hairstyle changes, distorted fingers, and local hair or eyebrow artifacts.

Editing examples demonstrate that the frame-aligned conditioning mechanism can preserve source actions while modifying appearance. In style transfer, the system applies the reference rendering style to the subject and background while preserving facial layout and object interactions. In virtual try-on, it maintains garment boundaries, material details, and hand–cloth occlusions. Subject replacement preserves source pose and interaction, and background replacement changes scene layout without removing the foreground action.

Figure 9

Figure 9

Figure 9: Avatar generation with cosmetics, hand interaction, and preservation of fine-grained character details.

Figure 10

Figure 10

Figure 10: Watercolor style transfer with preservation of subject structure and temporal attachment.

Figure 11

Figure 11

Figure 11: Virtual try-on under body motion and hand–garment interaction.

Figure 12

Figure 12

Figure 12: Character replacement while retaining source pose, interaction, and scene geometry.

Figure 13

Figure 13

Figure 13: Background replacement with preservation of foreground motion and camera structure.

These examples are useful for identifying failure modes, but they remain selectively presented. They cannot establish robustness across the full distribution of source videos, references, camera motions, occlusions, or user instructions.

Limitations and open questions

The paper leaves several methodological and empirical questions open. First, the ablations do not isolate the contributions of SRF, hybrid forcing, DPO, Streaming NFT, temporal dense captioning, background stabilization, the Refiner, and the VLM agent. Since Vidu S2 combines all of these components, the relative importance of each mechanism is unclear.

Second, the real-time claim is supported by frame-rate ranges and infrastructure descriptions, but the paper does not provide a complete latency–quality–cost characterization under a standardized hardware and service configuration. This matters particularly for Editing and spatial video, where interaction latency is perceptually more salient than in offline generation.

Third, the commercial-system comparisons use internal benchmarks, proprietary systems, and paired human judgments. The evaluators are trained and the presentation order is randomized, which strengthens the protocol, but the paper does not provide complete sample counts for every Avatar comparison, confidence intervals, inter-rater agreement, or statistical significance tests for each reported preference rate. Claims of 100% preference are therefore bounded by the corresponding, undisclosed or limited comparison sample sizes.

Fourth, the spatial-video component is evaluated qualitatively. The paper reports no stereo disparity accuracy, temporal left–right consistency, depth stability metric, motion-to-photon latency, headset-based comfort study, or comparison with dedicated stereo-generation methods. The proposed monocular-to-stereo conversion is consequently a feasibility pipeline rather than a validated high-fidelity spatial representation.

Finally, the system depends on extensive synthetic data generation, VLM captioning, model-based evaluators, and an agentic prompt-replanning layer. The paper does not fully characterize failure cases involving ambiguous instructions, incorrect VLM completion judgments, rapid state changes, multiple interacting objects, or contradictory references. Whether persistent state can remain reliable over substantially longer streams and more complex interactions remains unresolved.

Conclusion

Vidu S2 combines causal diffusion adaptation, Self-Replay Forcing, temporal data curation, preference optimization, low-latency inference engineering, and VLM-mediated interaction into a real-time video generation and editing system. Vidu S2-Avatar extends interactive character generation to 720p, dynamic references, full-body motion, and long-horizon audio–visual synthesis. Vidu S2-Editing applies frame-aligned diffusion editing to live streams across four practical task categories. Across public benchmarks and internal human evaluations, the system reports strong numerical advantages in perceptual quality, instruction adherence, temporal consistency, and virtual try-on quality (2609.11638). The paper’s principal unresolved issues concern component attribution, standardized latency reporting, statistical analysis of internal comparisons, and quantitative validation of spatial-video fidelity.

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

This paper introduces Vidu S2, a system for creating and editing video while people are watching and interacting with it.

Most video-generation tools work like this: you type a request, wait a long time, and then receive a finished video. Vidu S2 tries to work more like a live conversation. It can:

  • Create a digital character that responds to instructions.
  • Generate video in real time at up to 720p quality.
  • Change the character’s clothes, objects, or background during the video.
  • Edit a live video as it arrives.
  • Create video for virtual-reality headsets, where each eye sees a slightly different image.

Vidu S2 has two main parts:

  1. Vidu S2-Avatar: creates an interactive digital person or character.
  2. Vidu S2-Editing: changes an existing video in real time.

2. What questions are the researchers trying to answer?

The researchers are mainly asking:

  • Can a computer generate video quickly enough for people to interact with it naturally?
  • Can a digital character follow changing instructions, such as “dance,” “pick up this cup,” or “put on this hat”?
  • Can the system remember what happened earlier in a long video?
  • Can it edit a live video without changing the original person’s movements or timing?
  • Can the system change a person’s clothing, identity, style, or background while keeping the video smooth?
  • Can ordinary video be turned into spatial video for virtual reality?
  • Can all of this run on relatively affordable graphics cards instead of requiring extremely expensive computers?

In simple terms, the goal is to make video generation feel less like waiting for a movie to render and more like talking to a responsive animated character.

3. How did the researchers build and test the system?

Training with many kinds of video

The researchers trained Vidu S2 using a large collection of videos. These included:

  • Livestreams and talking-head videos
  • Films and television clips
  • Dance videos
  • 2D and 3D animations
  • Videos showing different body movements and facial expressions

Before using the videos, the researchers cleaned and organized them. They removed videos that were too blurry, unsafe, or badly damaged. They also measured more than just resolution. For example, a video labeled “1080p” might still look poor if it was heavily compressed.

The team also stabilized videos with moving cameras. This is similar to holding a shaky camera steady so that the computer can focus on the person’s movements instead of being confused by a moving background.

Describing what happens in each video

The system was given detailed descriptions of the videos. These descriptions explained events in time order, such as:

  1. A person reaches toward a cup.
  2. The person picks up the cup.
  3. The person smiles while holding it.

This is important because a live system must understand not only what happened, but also when it happened and what changed afterward.

A video model based on “diffusion”

Vidu S2 uses a type of artificial intelligence called a diffusion model. A simple way to imagine this is:

  1. The system starts with a very noisy picture or video.
  2. It gradually removes the noise.
  3. It turns the result into a clear video that matches the instructions.

Many diffusion systems create the whole video at once. Vidu S2 instead creates the video in small pieces, or segments, one after another. This makes it possible to show the first part while the next part is still being generated.

Training the system to work in a stream

A major challenge is that small mistakes can grow over time. For example, a character’s face might slowly change, or the background might begin to drift.

To reduce this problem, the researchers created a method called Self-Replay Forcing. In everyday language, the system:

  • Creates a long video by itself.
  • Looks back at that video.
  • Adds noise to parts of it.
  • Practices generating the parts again while checking how later mistakes are connected to earlier ones.

This is similar to a student reviewing a whole practice performance instead of studying only one sentence at a time.

The researchers also used:

  • Teacher forcing: training with the correct previous video frames.
  • Diffusion forcing: training with slightly noisy previous frames so the system learns to cope with imperfect results.
  • Preference training: showing the system which results people prefer, such as more natural movement or better lip synchronization.

Vidu S2-Avatar

For the digital-character system, users can provide:

  • A starting image of the character
  • Text or spoken instructions
  • New reference images during the video

A vision-LLM acts like a helper. It interprets the user’s request, writes a clearer instruction for the video generator, and checks whether the character completed the action.

For example, if the user says, “Pick up the blue cup and smile,” the helper tries to make sure the character continues holding the cup while smiling.

Vidu S2-Editing

The editing system receives a live video and an instruction. It can perform tasks such as:

  • Turning a real video into an anime or painting style
  • Changing a person’s clothing
  • Replacing the person with another character
  • Replacing the background

The system uses frame-aligned attention. This means that the edited version of a frame looks mainly at the matching original frame. It is like editing each frame while carefully keeping the same body position and timing.

This helps the edited person continue moving in the same way as the original person.

Making the system fast

The researchers used several computer-engineering tricks to reduce delay:

  • Faster ways to compare parts of images
  • Lower-precision numbers that require less computing power
  • Combining several small computer operations into one
  • Sharing work across multiple graphics cards
  • Reusing graphics-card memory when different parts of the system are not busy

These techniques are important because even a high-quality model would not be useful for live interaction if every frame took several seconds to produce.

Creating spatial video

Spatial video gives each eye a slightly different view. The brain combines these views to create a feeling of depth.

Vidu S2 estimates how far objects are from the camera and shifts them differently for the left and right eye. The result can be viewed through a virtual-reality headset.

For ordinary video, the system first edits or generates the video and then creates the two eye views. For existing stereo video, it edits the left and right views together.

4. What did the researchers find?

According to the paper, Vidu S2 improves on the earlier Vidu S1 system in several important ways.

Better real-time character generation

Vidu S2-Avatar can reportedly:

  • Generate video at 720p, compared with 540p for Vidu S1.
  • Produce about 25 to 42 frames per second, which is close to smooth video.
  • Follow more complex instructions, including large movements such as dancing.
  • Accept a new reference image at any moment during a stream.
  • Keep a character’s identity and appearance more stable over time.
  • Generate both the character’s video and audio together.

This means a character could respond to changing requests instead of being locked into one action from the beginning.

Real-time video editing

Vidu S2-Editing can change an incoming video while preserving the original movement and timing. The paper reports support for:

  • Style changes, such as turning a real scene into animation
  • Virtual try-on, such as changing someone’s clothes
  • Character replacement
  • Background replacement

The editing model was trained using about 800,000 videos, divided among these different tasks.

Better visual styles

For style transfer, the system was tested with many different appearances, including:

  • Cartoon or cel-shaded styles
  • Cyberpunk
  • Sketches
  • Monet-like paintings
  • Traditional Chinese painting styles

The researchers report that the system can handle more than 50 realistic and artistic styles.

Spatial video works in real time

The paper also shows that Vidu S2 can create or edit video for virtual-reality headsets. This could make a digital character appear to stand in front of the viewer rather than simply appear on a flat screen.

Comparison with other systems

The paper states that Vidu S2 performed better than the comparison systems, called baselines, in its experiments while still meeting the requirements for real-time use.

However, the provided paper text ends before the detailed experiment tables and numerical results. Therefore, the exact scores, testing conditions, and full comparisons are not included in the supplied material.

5. Why are these results important?

The main importance of this work is that it moves video generation toward live interaction.

With ordinary video-generation tools, users usually make a request, wait, and accept the final result. Vidu S2 aims to let users change their minds during the video. For example, someone could say:

  • “Make the character dance.”
  • “Now give the character a red jacket.”
  • “Put the character on a beach.”
  • “Make the scene look like a comic book.”

This could be useful for:

  • Interactive games
  • Virtual assistants
  • Livestreaming
  • Online education
  • Digital actors and entertainment
  • Virtual-reality experiences
  • Trying on clothes before buying them
  • Creative video editing

The system could also reduce the amount of traditional work needed to make animated characters and virtual worlds.

Limitations and future challenges

The paper also points out that the technology is not finished. Important challenges remain:

  • Spatial video needs very high resolution to look comfortable in a headset.
  • Even a small delay can feel unpleasant when a user moves their head.
  • Generated characters may still make mistakes or perform actions incorrectly.
  • Long videos must remain consistent without changing the character’s face or background.
  • The system still needs powerful computer hardware.
  • Converting a flat video into a believable 3D-like video can create holes or visual errors around object edges.

The researchers suggest future systems could create panoramic spatial worlds. In such a world, a user could turn their head and explore a scene that changes immediately in response to their actions.

Conclusion

Vidu S2 is a system designed to make AI-generated and AI-edited video work in real time. Its digital characters can respond to instructions, and its editing tools can change the appearance of live video without losing the original movement.

The paper’s biggest idea is that video generation should not only produce finished clips. It should also support continuous interaction, much like a conversation or a video game. If the remaining problems with speed, quality, and consistency are solved, systems like Vidu S2 could change how people create, watch, and experience digital video.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper leaves the following issues unresolved:

  • Incomplete experimental evidence: The provided paper text ends at the beginning of the experiments section, so the reported quantitative results, benchmark definitions, baselines, and ablation findings are unavailable for independent assessment.
  • Limited reproducibility: Key implementation details are unspecified, including model size, number of diffusion steps, segment length, cache-window length, training duration, optimization hyperparameters, and the exact SRF/DMD configuration.
  • Unclear real-time performance conditions: The claimed 25–42 FPS and real-time operation are not tied to sufficiently detailed hardware configurations, batch sizes, input/output resolutions, end-to-end latency measurements, or system load conditions.
  • Missing latency breakdown: The paper does not report the separate latency contributions of speech recognition, VLM prompt generation, model inference, super-resolution, decoding, spatial conversion, network transmission, and display.
  • No rigorous user-centered latency evaluation: It remains unknown whether the measured latency is low enough for natural interaction, especially for speech-driven control, rapid user actions, and camera passthrough applications.
  • Unclear quality–latency trade-offs: The paper does not quantify how much visual quality, temporal consistency, instruction following, and identity preservation degrade when latency, GPU count, precision, or attention sparsity is reduced.
  • Insufficient comparison with strong proprietary systems: The claim that Vidu S2 “outperforms all baselines” cannot be evaluated because the paper does not establish whether comparable commercial real-time systems, closed models, or alternative serving stacks were included.
  • No standardized benchmark for interactive streaming generation: The paper does not define a broadly reusable benchmark covering response latency, mid-stream instruction changes, action completion, long-horizon consistency, and user satisfaction.
  • Unresolved long-duration stability: Although the system is described as supporting continuous or infinite-length streams, the paper does not provide systematic measurements of drift, identity consistency, scene persistence, audio synchronization, or failure rates over hours-long sessions.
  • Limited analysis of error accumulation: The relative contributions of hybrid forcing, Diffusion Forcing, SRF, noisy caches, sink blocks, and sliding-window context to long-term stability are not isolated through comprehensive ablations.
  • Unclear SRF generality: It is not established whether Self-Replay Forcing improves performance across different segment lengths, motion types, resolutions, model scales, and domains, or whether its benefits depend on the specific Vidu architecture.
  • Potential training–inference mismatch remains: SRF detaches the original rollout and replays a fixed trajectory; the paper does not analyze whether this still leaves important distributional mismatches between replayed contexts and fully evolving inference-time contexts.
  • No assessment of failure recovery: The system’s behavior after severe drift, incorrect object interaction, malformed prompts, missing frames, dropped packets, abrupt scene changes, or temporary GPU overload is not described.
  • Limited coverage of user instructions: The claim of stronger instruction following is not broken down by instruction complexity, compositionality, negation, temporal ordering, contradictory commands, ambiguous language, or multi-step interactions.
  • Unclear action-state tracking: The agentic system is said to preserve states such as holding an object or wearing a hat, but the paper does not quantify state-tracking accuracy or test cases involving occlusion, object loss, multiple objects, or conflicting updates.
  • Dependence on external VLM and speech components: The paper does not report the computational cost, reliability, privacy implications, language coverage, or failure propagation associated with the VLM agent and speech-recognition pipeline.
  • No evaluation of interaction robustness across languages: It remains unknown whether text and speech control work reliably for different languages, accents, dialects, speaking rates, noisy environments, and code-switching.
  • Insufficient identity and appearance evaluation: The paper does not provide systematic metrics or human studies for preserving identity across dynamic reference changes, clothing changes, scene transitions, long streams, and large body motions.
  • Unclear scope of reference-image control: The system’s ability to interpret references for objects, clothing, backgrounds, and accessories is not evaluated across diverse viewpoints, lighting conditions, multiple subjects, low-resolution images, or references containing ambiguous content.
  • Editing data may inherit synthetic-model artifacts: Much of the editing training data is generated by existing video-editing models and filtered comparatively; the paper does not measure how artifacts, biases, or incorrect edits from those models affect Vidu S2.
  • Limited real-world editing evaluation: The four editing tasks are not sufficiently tested on uncontrolled footage containing occlusion, fast motion, camera shake, cuts, reflections, transparency, crowded scenes, unusual clothing, or severe illumination changes.
  • Frame-aligned attention may restrict legitimate temporal editing: The design enforces correspondence between each source and target frame, but the paper does not evaluate edits requiring temporal rearrangement, object insertion across frames, viewpoint changes, motion alteration, or content that appears at different times.
  • No quantitative preservation–editability analysis: The paper does not measure the balance between changing the requested content and preserving everything else, including background, facial identity, hand structure, motion, lighting, shadows, and audio.
  • Unclear support for editing audio: Vidu S2-Editing is described primarily as a visual editor, and the paper does not clarify whether speech, ambient sound, lip synchronization, or audio effects are preserved or modified.
  • Insufficient temporal-consistency metrics for editing: The paper does not report detailed measurements of flicker, texture swimming, identity switching, boundary instability, temporal warping, or consistency across chunk boundaries.
  • Unknown generalization beyond the selected styles and tasks: The claim of generalization to more than 50 styles does not establish performance on styles absent from training, mixed styles, user-defined styles, or styles with strong geometric and semantic transformations.
  • No fairness or demographic robustness analysis: Performance across skin tones, ages, genders, body types, disabilities, hairstyles, cultural clothing, and non-human characters is not reported.
  • Safety and misuse risks are underexplored: The paper does not evaluate impersonation, non-consensual identity replacement, deceptive livestreaming, sexualized edits, political manipulation, copyrighted references, or safeguards against harmful instructions.
  • Privacy implications are unspecified: The handling, retention, and processing of user-uploaded faces, voices, personal videos, and reference images are not discussed.
  • Spatial-video conversion is only approximately 3D: The monocular-to-stereo pipeline relies on estimated depth and horizontal disparity, but the paper does not quantify depth accuracy, geometric correctness, occlusion handling, or view consistency.
  • No perceptual evaluation in head-mounted displays: The spatial-video claims lack user studies measuring immersion, visual comfort, cybersickness, vergence–accommodation conflict, depth plausibility, and presence.
  • Unresolved stereo consistency problems: For stereoscopic editing, the paper does not report how it prevents left–right identity mismatches, inconsistent occlusion boundaries, unequal texture details, vertical disparity, or temporal divergence between views.
  • Insufficient resolution and field-of-view analysis for VR: The paper acknowledges that spatial video needs higher resolution but does not specify the supported headset resolutions, field of view, refresh rates, display pipelines, or quality degradation under those conditions.
  • No rigorous end-to-end passthrough evaluation: The proposed camera-based spatial editing scenario is not tested for motion-to-photon latency, head-motion compensation, frame drops, synchronization errors, or user safety.
  • Panoramic and viewpoint-dependent generation remains speculative: The discussion proposes panoramic spatial video and freely explorable scenes, but the current method does not address view-conditioned generation, multi-view consistency, scene geometry, or persistent world state.
  • Resource scalability is unclear: The paper claims deployment on low-cost GPUs but does not report cost per stream, maximum concurrent users, energy consumption, thermal constraints, or scaling behavior under multi-user workloads.
  • Multi-GPU communication overhead is not quantified: The benefits and limitations of quantized context-parallel communication are not separated from computation speedups, particularly for different numbers of GPUs and network interconnects.
  • Quantization and approximate-attention failure cases are undocumented: The paper does not identify which visual phenomena, motion patterns, resolutions, or layers are most vulnerable to W8A8 quantization and sparse/approximate attention.
  • Training-data distribution and licensing are unspecified: The sources, demographic composition, copyright status, consent procedures, and geographic or cultural coverage of the video and audio data are not disclosed.
  • Caption quality is insufficiently validated: The proposed temporally dense, multi-agent captions are described conceptually, but their factual accuracy, temporal-boundary precision, hallucination rate, and effect on downstream performance are not fully demonstrated.
  • No analysis of data-filtering bias: High-clarity selection, background stabilization, shot filtering, and safety filtering may remove certain cameras, environments, motion types, or demographic groups; the consequences of these selection biases are not examined.
  • Human preference and reward-model validity are unclear: The paper does not describe annotator populations, agreement rates, reward-model calibration, preference criteria, or whether preference optimization improves objective correctness versus superficial visual appeal.
  • No comprehensive ablation of data versus architecture: It remains unclear whether the reported gains come primarily from additional dance and animation data, improved captions, new training objectives, inference optimizations, the Refiner, or the agentic system.
  • Failure rates are not reported: The paper does not provide an operational taxonomy or frequency estimates for severe artifacts, identity collapse, incorrect edits, instruction refusal, temporal desynchronization, spatial discomfort, and system crashes.
  • Robustness to distribution shift is unknown: Performance on unseen cameras, frame rates, codecs, lighting, environments, body poses, animation styles, and user behaviors is not established.
  • The relationship between visual quality and semantic correctness is unresolved: The paper claims improvements in naturalness and instruction adherence, but it does not clarify whether evaluations distinguish photorealistic quality from actually performing the requested action or edit.
  • No longitudinal user study is provided: It remains unknown whether users find continuous interaction more useful, engaging, controllable, or trustworthy than conventional offline video-generation workflows over extended use.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Interactive digital-character assistants for customer service, education, and entertainment — Sector: software, education, media, customer support. Vidu S2-Avatar can support real-time characters that respond to text or speech, change expressions and poses, and perform actions such as dancing or handling referenced objects. This could be deployed in virtual receptionists, language-learning tutors, museum guides, livestream hosts, and interactive game characters. Dependencies: Reliable speech recognition, moderation, identity and voice-consent procedures, sufficient GPU capacity, and safeguards against misleading anthropomorphic behavior.
  • Real-time livestream and social-media production — Sector: media, marketing, creator tools. Vidu S2-Editing can apply style transfer, background replacement, character replacement, and virtual try-on directly to an incoming video stream. A creator workflow could include live anime rendering, branded visual effects, automatic studio backgrounds, or real-time replacement of a performer with an approved avatar. Dependencies: Stable input video, low-latency network connectivity, acceptable temporal consistency, platform support for GPU-accelerated processing, and clear disclosure that content is AI-generated or modified.
  • Virtual try-on for apparel and accessories — Sector: retail, fashion, advertising, e-commerce. Retailers can integrate reference images of clothing into live video so that a user sees an edited version of their appearance while moving naturally. The same workflow could support hats, glasses, jewelry, uniforms, and promotional costumes. Dependencies: Accurate body and garment tracking, preservation of product details and sizing information, representative training data across body types and skin tones, and the understanding that visual results are not necessarily physically accurate fits.
  • Live background and scene replacement — Sector: broadcasting, remote work, events, real estate. The system can replace backgrounds while preserving the subject’s motion and timing. Potential tools include virtual production for news and livestreams, privacy-preserving backgrounds for teleconferencing, remote-event staging, and real-time real-estate visualization. Dependencies: Adequate foreground segmentation, lighting compatibility, prevention of accidental removal of important objects, and compliance with workplace or broadcast disclosure rules.
  • Real-time character and identity replacement for post-production previews — Sector: film, television, advertising, games. Production teams can preview alternative characters, costumes, visual styles, or performers while filming, rather than waiting for offline rendering. This can accelerate storyboarding, casting visualization, advertising concept development, and live creative direction. Dependencies: Final production still requires higher-fidelity rendering and human review; use of a person’s likeness requires consent and licensing; generated previews may not preserve exact facial identity or physical interaction.
  • Interactive avatar-based training and simulation — Sector: corporate training, education, healthcare simulation, public safety. A reference-conditioned avatar can act as a patient, customer, instructor, or emergency scenario participant and respond to spoken instructions. Dynamic references could introduce objects, uniforms, locations, or props during a session. Dependencies: Domain-specific validation, carefully authored scenarios, low hallucination rates, accessibility support, and the requirement that generated behavior not be treated as professional medical, legal, or safety advice.
  • Automated video-quality and dataset-preparation pipelines — Sector: academia, AI engineering, media archives. The paper’s high-clarity selection, cut-point detection, background stabilization, temporal captioning, and multi-stage filtering methods can be used independently to curate training or production datasets. Organizations could build pipelines that score compression artifacts, detect scene cuts, stabilize moving-camera footage, and produce event-ordered captions. Dependencies: Access to vision-language and speech models, domain-specific quality thresholds, annotation verification, copyright clearance, and controls for demographic and cultural bias.
  • Low-cost deployment of real-time generative video services — Sector: cloud infrastructure, software engineering. The inference innovations—quantized W8A8 matrix multiplication, efficient and sparse attention, kernel fusion, CUDA Graphs, inter-module scheduling, and multi-GPU context parallelism—can be incorporated into serving stacks for real-time video generation or editing. These techniques may reduce per-session GPU cost and make interactive video feasible on consumer or lower-cost data-center hardware. Dependencies: Hardware-specific kernel engineering, quality testing under quantization, memory bandwidth, multi-GPU communication, and workload patterns that are sufficiently stable to benefit from graph capture and scheduling.
  • Live visual personalization in daily communication — Sector: consumer applications. Users could modify clothing, scenery, art style, or avatar identity during video calls and live messaging. Dynamic reference updates would allow a user to introduce a new background, accessory, or visual theme without restarting the session. Dependencies: Consumer-device performance, privacy-preserving processing, user control over retention of video and reference images, and protections against impersonation or non-consensual appearance manipulation.

Long-Term Applications

  • Immersive VR companions and interactive spatial characters — Sector: VR, gaming, entertainment, social platforms. The avatar pipeline can generate synchronized left- and right-eye views for head-mounted displays, enabling conversational characters that appear situated in the user’s space. Possible products include VR tutors, game companions, virtual performers, and social avatars that react continuously to speech and visual references. Dependencies: The paper identifies unresolved requirements for higher resolution, lower end-to-end latency, depth stability, disocclusion handling, and prevention of motion sickness. Reliable head-pose synchronization and robust identity consistency are also necessary.
  • Spatial-video editing for passthrough and augmented reality — Sector: AR/VR, robotics, remote collaboration. Real-time editing of monocular or stereoscopic camera streams could enable live replacement of objects, clothing, characters, and backgrounds in headset passthrough. This may support privacy masking, remote assistance, immersive shopping, virtual stage design, and collaborative visualization. Dependencies: Accurate stereo consistency, geometric alignment between views, low latency during head movement, robust occlusion handling, and safety mechanisms preventing edited imagery from obscuring hazards in the physical environment.
  • Panoramic, interactive generated worlds — Sector: games, metaverse platforms, simulation, architecture. The paper proposes extending fixed-view spatial video to panoramic spatial video. A mature version could generate scenes, characters, and events as users turn their heads and interact, reducing the need for manually authored 3D models, materials, animations, and environments. Dependencies: View-consistent generation across wide fields of view, persistent world state, multi-user synchronization, six-degree-of-freedom interaction, high-resolution rendering, and substantially lower latency than demonstrated for ordinary monocular streaming.
  • Real-time virtual production and adaptive broadcasting — Sector: film, television, sports, advertising. Broadcasters could change visual style, locations, costumes, presenters, or advertising elements while a live event is underway. For example, the same sports broadcast could provide different sponsor environments or stylistic feeds for different audiences. Dependencies: Broadcast-grade reliability, deterministic temporal continuity, rights management, editorial approval, provenance tracking, and failure-safe fallbacks when the model produces artifacts.
  • Embodied robotics and telepresence interfaces — Sector: robotics, healthcare, industrial operations. The model’s ability to follow sequential instructions, preserve state across actions, and incorporate new reference images could support visual interfaces for teleoperated robots or expressive robotic avatars. A system might show a remote operator how a robot will handle an object, change its simulated environment, or communicate task progress through a generated character. Dependencies: Vidu S2 generates visual content rather than verified robot control commands. Safe deployment would require grounding in robot state, calibrated perception, deterministic planning, latency guarantees, and human confirmation before physical actions.
  • Clinical and therapeutic visualization tools — Sector: healthcare, rehabilitation, mental-health support. Interactive avatars could provide guided rehabilitation demonstrations, patient education, language-accessible explanations, or simulated clinician-patient interactions. Spatial video could eventually make exercises and anatomical demonstrations more immersive. Dependencies: Clinical validation, medical-device and healthcare regulation, privacy protection, medically accurate content, accessibility, and strict separation between generated educational material and diagnostic or treatment decisions.
  • Personalized education and language learning at scale — Sector: education, tutoring, accessibility. A tutor avatar could react to spoken commands, demonstrate gestures or dances, change clothing and settings to reflect cultural contexts, and introduce objects through reference images. Temporally ordered captions and event-aware prompting could support lessons involving sequential actions. Dependencies: Curriculum alignment, factual accuracy, age-appropriate moderation, teacher oversight, multilingual speech quality, equitable access to compute, and evaluation of learning outcomes rather than visual appeal alone.
  • Real-time digital humans for finance, government, and public services — Sector: finance, policy, civic technology. Government agencies, banks, and public institutions could deploy multilingual interactive agents for navigating forms, explaining procedures, or simulating policy scenarios. Dynamic references could show relevant documents, locations, or objects during an interaction. Dependencies: High standards for factuality, auditability, accessibility, data protection, authentication, human escalation, and explicit labeling of synthetic agents. Generative outputs should not independently authorize financial or governmental decisions.
  • Policy and standards for synthetic live media — Sector: policy, law, platform governance. The real-time nature of Vidu S2 makes provenance mechanisms, watermarking, consent standards, and disclosure policies especially important. Regulators and platforms could establish requirements for labeling live appearance manipulation, licensing likenesses, retaining generation logs, and reporting model failures. Dependencies: Interoperable provenance standards, enforceable consent and copyright frameworks, protection against evasion, and policies that distinguish benign personalization from fraud, impersonation, or political deception.
View Paper Prompt  View All Prompts 

Glossary

  • Autoregressive rollout: Sequential generation in which each output segment becomes context for generating the next segment. “After the student performs a long autoregressive rollout, we take the entire student-generated trajectory”
  • Background stabilization: Processing that compensates for camera-induced background motion to produce a visually stable scene. “we introduce a background stabilization operator”
  • Bidirectional attention: Attention mechanism allowing tokens to interact across both earlier and later positions in a sequence. “The generated-video tokens interact bidirectionally across the target sequence”
  • Causal attention mask: Attention constraint that prevents a model from using future sequence elements. “we replace its bidirectional temporal attention with a block-wise causal attention mask”
  • Causal adaptation: Conversion of a non-streaming model into one that generates sequentially using only valid historical context. “This causal adaptation equips the model with an initial capability for streaming video-audio generation”
  • CUDA Graphs: NVIDIA CUDA mechanism for capturing and replaying a sequence of GPU operations with reduced launch overhead. “Second, we use CUDA Graphs for stable execution sequences.”
  • Disoccluded regions: Areas newly revealed when a viewpoint changes and that were hidden in the original view. “fill holes in disoccluded regions”
  • Diffusion Forcing: Training approach that adds independently sampled noise to historical sequence states to improve robustness during autoregressive generation. “Under Diffusion Forcing, noise is injected into the historical states at sampled noise levels”
  • Diffusion Transformer (DiT): Transformer architecture adapted to perform denoising in diffusion-based generative models. “Vidu S2-Editing employs a Diffusion Transformer (DiT) for instruction-guided video editing”
  • Direct Preference Optimization (DPO): Preference-learning method that directly optimizes a model using preferred and dispreferred outputs without separately fitting a reward model. “we use diffusion-based Direct Preference Optimization (DPO)”
  • DMD supervision: Supervision based on Distribution Matching Distillation, which trains a student generator to match a teacher’s output distribution. “DMD supervision is applied to all blocks within the replayed segment.”
  • Dynamic reference: Reference input that can be introduced or changed while generation is already underway. “generation with dynamic references that can be updated at any moment”
  • Embedding: Numerical representation of data, such as text, images, or video, used by a model for computation. “Clipping, Filtering, Speech Processing, Captioning, and Embedding”
  • Frame-aligned attention: Attention mechanism that restricts each generated frame’s interaction with a source frame at the same temporal position. “The key design is frame-aligned attention”
  • Geometric transformation: Mathematical operation that changes an image’s spatial arrangement, such as rotation, translation, or scaling. “background motion to be approximated by geometric transformations across frames”
  • GEMM: General Matrix-Matrix Multiplication, a highly optimized matrix operation central to neural-network computation. “per-block W8A8 GEMM”
  • Gradient flow: Propagation of derivatives through a computation graph during neural-network training. “allowing the loss of a later segment to propagate to preceding segments during the replay pass”
  • Hybrid forcing: Training strategy combining clean historical contexts with deliberately noised historical contexts. “We adopt a hybrid training strategy that combines Teacher Forcing and Diffusion Forcing”
  • Image-to-video (I2V): Generation of a video conditioned on an input image, often used as its initial frame. “For I2V training, r\bm{r} represents the first frame of the target video”
  • Inference latency: Time required for a model to produce an output during deployment. “These optimizations preserve generation quality while improving inference efficiency and overall resource utilization, enabling real-time, low-latency inference services.”
  • Kernel fusion: Combining multiple GPU operations into a single kernel to reduce memory traffic and launch overhead. “First, we fuse adjacent operations into custom Triton/CUDA kernels.”
  • KV cache: Stored key and value representations from attention computation that are reused across sequential decoding steps. “The original rollout and its KV caches are detached before replay”
  • Latent space: Compressed representational space in which generative models perform transformations or diffusion operations. “we introduce a one-step super-resolution refiner that operates directly in the latent space”
  • Low-bit quantization: Representing model values with fewer bits to reduce memory use and computation. “It uses SageAttention, SpargeAttention, and Sparse-Linear Attention, low-bit GEMM”
  • Monocular video: Video containing a single view rather than separate views for the two eyes. “Vidu S2-Avatar can already generate monocular video in real time”
  • Mode collapse: Generative-model failure in which outputs become insufficiently diverse. “we add a perceptual loss to the replayed student outputs to mitigate mode collapse”
  • On-policy: Based on trajectories generated by the current model under its own inference procedure. “Here, ``on-policy'' refers to the autoregressive trajectory and historical contexts generated by the current student model under its inference procedure.”
  • Parallax: Apparent relative motion between foreground and background objects caused by viewpoint or camera movement. “The resulting parallax is typically small”
  • Perceptual loss: Training loss that compares high-level visual features or perceptual similarity rather than only individual pixels. “we add a perceptual loss to the replayed student outputs”
  • Quantization: Reduction of numerical precision used to represent model parameters or activations. “Per-tensor and per-channel quantization methods are faster”
  • Reference-conditioned generation: Generation controlled by a provided image or other reference that specifies desired appearance or identity. “Beyond training on data built for reference-conditioned generation”
  • RoPE encoding: Rotary positional encoding, which represents token positions by rotating components of their representations. “Condition-specific RoPE encodings represent the spatial and temporal positions of the different token streams.”
  • Self-Replay Forcing (SRF): Training method that re-noises a model’s own generated trajectory and replays it with gradient-enabled connections. “We introduce Self-Replay Forcing (SRF), an on-policy DMD”
  • Stereoscopic video: Video containing distinct, coordinated views for the left and right eyes to convey depth. “For stereoscopic input, it can jointly edit paired views”
  • Super-resolution: Increasing spatial resolution while reconstructing or synthesizing additional fine-grained detail. “To recover fine-grained spatial details from the low-resolution latent outputs of the causal backbone”
  • Temporal dense captioning: Captioning that describes events, actions, and outcomes in chronological order with temporal boundaries. “Vidu S2 adopts temporally ordered dense captions.”
  • Teacher Forcing: Training procedure that conditions sequence prediction on ground-truth previous states rather than the model’s own outputs. “Under Teacher Forcing, the model is conditioned on clean ground-truth historical states.”
  • Virtual try-on: Image or video synthesis task that digitally places clothing or accessories onto a person. “Vidu S2-Editing supports editing a video stream in real time, including style transfer, virtual try-on, character replacement, and background replacement.”
  • Vision-LLM (VLM): Model that jointly processes visual and linguistic information. “A vision-LLM (VLM) agent uses these inputs to generate prompts”
  • W8A8: Quantization format using 8-bit weights and 8-bit activations. “per-block W8A8 GEMM”
  • Warping: Geometric remapping of image pixels to simulate a changed viewpoint or spatial transformation. “it warps the image in opposite horizontal directions to synthesize the left and right views”
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Tweets

Sign up for free to view the 6 tweets with 951 likes about this paper.

Sign Up for Free