LOTUS: Hybrid Video Summarization

• LOTUS is an interactive system that blends abstractive summarization with selective extractive clips to create short-form social videos.
• It uses script-driven generation, visual concept extraction, and weighted clip matching based on speech, keyframe, and position metrics.
• The hybrid approach effectively balances compression and authenticity, streamlining the repurposing of long-form content for social media.

LOTUS is an interactive system for turning a long-form video into a short-form social video by combining abstractive and extractive summarization (Barua et al., 10 Feb 2025). It is designed for the production of short-form videos in styles associated with TikTok, Instagram, and Shorts, where creators must compress longer source material while preserving coherence, visual engagement, and authenticity. Its central premise is that neither purely extractive nor purely abstractive summarization is sufficient on its own: extractive summaries preserve the original audio-visual relationship but are constrained by the source material’s existing clip structure, whereas abstractive summaries offer greater compression and reorganization flexibility but make narration and visual matching more difficult. LOTUS addresses this tradeoff by generating an initial abstractive short video and then selectively incorporating extractive clips from the original video where preserving the creator’s speech, realism, or synchronized audiovisual delivery is beneficial (Barua et al., 10 Feb 2025).

1. Conceptual basis and problem formulation

LOTUS is framed around a recurrent production problem in short-form video editing: creators repurpose long-form videos into much shorter outputs, but must simultaneously identify salient moments, preserve or reconstruct a coherent story, compress content to social-media length, maintain strong hooks and endings, avoid awkward cuts, and sometimes decouple audio and visuals so narration remains concise while visuals stay interesting (Barua et al., 10 Feb 2025). In genres such as tours, travel, reviews, and tutorials, important information is often distributed across the source video and conveyed jointly through narration and visuals. Editors in the formative study described this process as taking hours to days, largely because of clip selection and sequencing (Barua et al., 10 Feb 2025).

The system distinguishes two modes of short-form generation. Extractive short videos are composed of clips directly taken from the long-form source. Abstractive short videos introduce newly generated narration and then pair that narration with visuals drawn from the source video. LOTUS does not treat these as mutually exclusive alternatives. Instead, it treats them as complementary mechanisms whose relative advantages depend on the structure of the source material and the desired editorial effect (Barua et al., 10 Feb 2025).

This design leads to a specific interpretation of short-form creation: not as a single summarization problem, but as a coordination problem across script compression, shot selection, audiovisual alignment, and authenticity management. A plausible implication is that LOTUS should be understood less as a conventional summarizer than as a hybrid authoring system in which automated summarization and editorial control are deliberately interleaved.

2. Abstractive summary video generation

The first stage of LOTUS creates an abstractive summary video from the long-form input (Barua et al., 10 Feb 2025). Starting from the long-form transcript, LOTUS prompts GPT-4o to generate a short-form transcript in the style of the original speaker, explicitly encouraging a strong social-media-friendly hook and reuse of the original introduction and conclusion. To better preserve visual coverage, LOTUS extracts noun phrases from the long-form transcript using spaCy and includes them in the prompt as candidate visuals (Barua et al., 10 Feb 2025).

After generating the short-form transcript, LOTUS identifies visual concepts: concrete words or phrases in the new script that should correspond to shots. This is done by scene-detecting the long video into clips, generating clip descriptions with GPT-4o, and prompting GPT-4o to embed suitable visual concepts directly into the short-form sentences (Barua et al., 10 Feb 2025). The source video is therefore not matched to narration only after summarization; visual anchoring is introduced into the script generation process itself.

Clip selection is then handled as a weighted matching problem. Each long-form clip is scored against a target visual concept using four signals:

1. Speech similarity between the clip transcript and the visual concept, using nomic-embed-text embeddings.
2. Keyframe similarity between the clip’s keyframe and the visual concept, using CLIP.
3. GPT-4o scoring, where GPT-4o rates a filtered set of about 25 candidate clips given their keyframes and speech.
4. Position-based alignment, which rewards clips appearing at roughly the same relative point in the source video as the concept appears in the generated summary (Barua et al., 10 Feb 2025).

The position score is given as

$$Score_{pos} = 1 - Pos(visual\ concept) - Pos(long-form\ clip)$$

where Pos(visual concept) is the concept’s normalized position in the short-form transcript and Pos(long-form clip) is the candidate clip’s normalized position in the long-form video (Barua et al., 10 Feb 2025). After scoring, LOTUS assigns the highest-scoring long-form clip to each visual concept. If the same source clip is selected multiple times, the system iteratively replaces duplicates with the next-best alternatives until duplicates are removed (Barua et al., 10 Feb 2025).

The script is then synthesized into narration using ElevenLabs, and the matched clips are aligned to the generated speech. LOTUS adjusts clip playback speed when necessary so clip duration fits narration duration, producing the initial abstractive short video shown to the user (Barua et al., 10 Feb 2025).

This stage is notable because the abstraction is script-driven rather than clip-driven. The short-form video is first defined as a compressed narrative and only then grounded in source visuals. This suggests that LOTUS treats temporal structure and rhetorical framing—especially the hook and conclusion—as primary organizational units, with clip retrieval serving that structure rather than determining it.

3. Blended abstractive-extractive construction

The second stage produces a blended abstractive-extractive video by selectively replacing some generated-narration segments with extractive segments from the original video (Barua et al., 10 Feb 2025). The rationale is that certain moments benefit from preserving the original creator’s voice, camera-facing delivery, or naturally synchronized audiovisual pairing.

To do this, LOTUS segments both the generated abstractive script and the long-form transcript. The abstractive script is segmented with GPT-4o; the long-form transcript is segmented using ROPE’s segmentation algorithm (Barua et al., 10 Feb 2025). It then compares every abstractive segment to extractive candidates using four metrics:

1. Speech similarity between abstractive and extractive segments.
2. Visual connection, defined as the similarity between a segment’s speech and its keyframe via CLIP.
3. Coverage, the number of noun phrases present in the segment speech, extracted by spaCy.
4. Position-based alignment, based on relative positions of abstractive and extractive segments (Barua et al., 10 Feb 2025).

If the difference between an abstractive segment and its highest-scoring extractive alternative is below an empirically chosen threshold, LOTUS keeps both as plausible options and generates permutations of the video using those choices. Among these candidate mixtures, it selects the one with the highest coherence score, using GPT-3 loss as the coherence criterion (Barua et al., 10 Feb 2025). The selected sequence becomes the final blended short-form video. LOTUS then applies volume normalization and denoising so audio sounds consistent across generated and extracted segments (Barua et al., 10 Feb 2025).

The resulting end-to-end workflow is:

input long-form video → scene detection and transcript extraction → noun phrase extraction → GPT-4o short-form transcript generation → visual concept extraction → scoring and alignment of long-form clips to summary concepts → generated narration + clip timing adjustment to form initial abstractive result → segmentation of both summary and source transcripts → candidate extractive replacement scoring → permutation search and coherence optimization → final blended short video (Barua et al., 10 Feb 2025).

This architecture makes an important distinction between two kinds of fidelity. One is content fidelity, preserved by the script-driven abstractive summary and source-clip matching. The other is performative fidelity, restored when extractive segments preserve the original creator’s voice and audiovisual timing. LOTUS operationalizes both.

4. Interaction model and editing interface

The interaction model is central to LOTUS. After the automated pipeline produces a draft, the user edits and refines it through an interface with three panes: a video player pane, a long-form clips pane, and a short-form clips pane (Barua et al., 10 Feb 2025). The long-form pane presents source clips in grid or list form, with keyframes and optionally speech, allowing users to skim the source at clip granularity rather than scrubbing a full timeline. It also supports CLIP-based search by text prompt or by selecting a keyframe, reordering clips by similarity (Barua et al., 10 Feb 2025).

The short-form pane shows the current summary clips and supports direct manipulation. Creators can refine the generated result in several ways. They can toggle any clip between abstractive and extractive modes. In abstractive mode, they can edit narration text, generate alternative text, and choose a speaker voice from diarized speakers found via PyAnnote or from a default voice. In extractive mode, they can preserve source audio and optionally denoise it (Barua et al., 10 Feb 2025).

The editing operations include dragging source clips into the short-form sequence, reordering clips, deleting clips, trimming or extending clips, and replacing a clip with one of 20 suggested alternative clips that better match the same transcript section (Barua et al., 10 Feb 2025). LOTUS also provides an align feature that maps a short-form clip back to its location in the long-form clip list and shows nearby context, enabling users to pull in surrounding clips or borrow wording from adjacent transcript sections (Barua et al., 10 Feb 2025).

The system’s design choices follow three stated goals from the formative work: help users identify key clips, help them arrange clips into coherent sequences, and let them flexibly decouple and recombine audio and visuals (Barua et al., 10 Feb 2025). This led to several concrete interface decisions: beginning from a generated short-form structure rather than a blank long video, representing the source as clips with keyframes and captions, allowing local rewriting of narration, exposing search and alternative clip suggestions, and preserving opportunities to use the original creator’s voice for authenticity (Barua et al., 10 Feb 2025).

The implementation uses a React frontend, Flask backend, and a separate Flask API for logging; video and audio processing use MoviePy and torchaudio (Barua et al., 10 Feb 2025).

5. Prompting, segmentation, and algorithmic control surfaces

Although LOTUS is presented as a video editing system, much of its algorithmic behavior is determined by prompt design rather than by closed-form optimization objectives. The paper explicitly lists several important prompts: one for short-form transcript generation in TikTok/Instagram/Shorts style, one for visual concept extraction that embeds <VIS> ... </VIS> spans into sentences, one for multimodal GPT-4o clip scoring from 0 to 1, one for transcript segmentation into <TITLE> and <SEG> blocks, and one for generating alternative speech for a selected shot given before/after context and shot duration (Barua et al., 10 Feb 2025).

These prompts are not incidental interface features. They define the summary script, visual anchors, segmentation structure, clip scores, and editable rewrites. In that sense, LOTUS uses LLM prompting as a control mechanism inside the summarization pipeline itself (Barua et al., 10 Feb 2025).

This has two consequences. First, the system’s behavior is partly contingent on language-model instruction following rather than solely on deterministic ranking heuristics. Second, the editable draft becomes an important stabilizing layer: creators can revise generated text, swap in extractive material, and adjust clip structure after the LLM-defined initial pass. A plausible implication is that the interface is not merely post-processing; it is part of the reliability strategy.

6. Evaluation, genre effects, and authorship implications

The evaluation has two parts: a results evaluation comparing generated videos, and a user study comparing editing workflows (Barua et al., 10 Feb 2025).

In the results evaluation, the authors compared three methods: an extractive baseline adapted from ROPE, LOTUS’s abstractive method, and LOTUS’s blended abstractive-extractive method (Barua et al., 10 Feb 2025). They selected six YouTube videos from trending content, each 5–20 minutes long and visually diverse, excluding podcasts, TED talks, and lectures. The videos covered travel, cooking, spotlight/interview, and home tour content, with source durations from 7:25 to 17:18 and generated short versions roughly 50 seconds to 1:25 (Barua et al., 10 Feb 2025). Twelve annotators familiar with short-form video ranked outputs by overall preference and rated hook, narrative, and visual appeal on 5-point Likert scales (Barua et al., 10 Feb 2025).

The blended method had the best average overall ranking:

• mixed: $p = 1.88, \sigma = 0.72$
• abstractive: $p = 2.06, \sigma = 0.77$
• extractive: $p = 2.04, \sigma = 0.82$

but the difference was not statistically significant (Barua et al., 10 Feb 2025). More informative than the aggregate ranking was the genre pattern. Extractive and mixed outputs were preferred for videos where a narrator was prominently on camera, because preserving the source speaker increased authenticity and engagement. Abstractive outputs were preferred when the source did not strongly depend on a visible narrator, because generated narration made it easier to cover more content concisely (Barua et al., 10 Feb 2025). Missing hooks and conclusions were especially damaging for extractive methods; in some cases the mixed method prevailed because it combined an abstractive storyline with an extracted opening hook (Barua et al., 10 Feb 2025).

The user study used a within-subjects design with 8 participants who had video editing experience, recruited via Upwork and university mailing lists. Participants averaged 22 years old and reported 3.71 years of editing experience on average (Barua et al., 10 Feb 2025). They used both LOTUS and their preferred existing editing tool, such as Adobe Premiere Pro or CapCut, on two selected long-form YouTube videos of similar length and visual diversity, with up to 20 minutes per task after 10 minutes of familiarization per source video (Barua et al., 10 Feb 2025).

The findings were positive for LOTUS as an authoring aid. All participants said they would want to use it in the future. They produced short-form videos they considered comparable in quality to those made with their usual tools, without increased mental demand, and generally liked the editing process in LOTUS more (Barua et al., 10 Feb 2025). Several participants valued the fact that LOTUS starts with a coherent draft rather than requiring them to cut down from the full source. The initial result was described as especially useful as a structure or “base to go off of” (Barua et al., 10 Feb 2025). Common interactions included align, drag shot, delete shot, trim shot, and render, along with search, text editing, and alternative clip replacement (Barua et al., 10 Feb 2025).

The qualitative findings also clarify how creators divided labor between clip types. Abstractive clips were valued for flexibility, because users could say exactly what they wanted, shorten long narration, and pair preferred visuals with revised speech. Extractive clips were valued for credibility, personality, and naturalness, because the original creator’s voice and planned audiovisual delivery felt more authentic (Barua et al., 10 Feb 2025).

These results complicate any simple claim that one summarization mode is superior. LOTUS’s empirical contribution is not that hybridity always wins numerically, but that hybridization exposes a controllable tradeoff between compression and authenticity that varies by genre and presentation style.

7. Limitations and future directions

The paper identifies several limitations. Generated speech sometimes sounded unnatural or lacked expressive nuance, especially with multiple speakers, reducing the appeal of abstractive clips (Barua et al., 10 Feb 2025). Performance also depends on video type: instructional videos are challenging because they require precise step-by-step audiovisual alignment, and existing source visuals may not adequately cover a concise rewritten script (Barua et al., 10 Feb 2025).

Participants additionally noted interface limitations relative to mature editors: no traditional horizontal timeline, limited trimming granularity, latency before hearing regenerated speech, and fewer low-level controls such as independently editing audio and video channels or adjusting aspect ratio (Barua et al., 10 Feb 2025). These observations matter because LOTUS is presented not as a replacement for full-featured editing software, but as a hybrid human-AI video editing system (Barua et al., 10 Feb 2025).

The authors suggest future work on better hooks and conclusions, B-roll recommendation or generation, more expressive speech synthesis, natural-language editing commands, multiple starting drafts, and integration with existing editing tools (Barua et al., 10 Feb 2025). This suggests an ongoing shift from summarization-as-output toward summarization-as-editable-structure.

A common misconception would be to treat LOTUS as a fully automatic short-video generator. The paper explicitly positions it otherwise: its core contribution is showing that short-form creation benefits from starting with an abstractive, script-driven summary for compression and coverage, then selectively recovering extractive source clips to restore authenticity, natural audiovisual alignment, and engaging moments (Barua et al., 10 Feb 2025). In that formulation, automation provides a strong draft, but authorship remains distributed across generation, ranking, and human refinement.