---
title: 'LOTUS: Hybrid Video Summarization'
url: https://www.emergentmind.com/topics/lotus
type: topic
---

# LOTUS: Hybrid Video Summarization

LOTUS is an interactive system for turning a long-form video into a short-form social video by combining **abstractive** and **extractive** summarization [2502.07096]. It is designed for the production of short-form videos in styles associated with TikTok, Instagram, and Shorts, where creators must compress longer source material while preserving coherence, visual engagement, and authenticity. Its central premise is that neither purely extractive nor purely abstractive summarization is sufficient on its own: extractive summaries preserve the original audio-visual relationship but are constrained by the source material’s existing clip structure, whereas abstractive summaries offer greater compression and reorganization flexibility but make narration and visual matching more difficult. LOTUS addresses this tradeoff by generating an initial abstractive short video and then selectively incorporating extractive clips from the original video where preserving the creator’s speech, realism, or synchronized audiovisual delivery is beneficial [2502.07096].

## 1. Conceptual basis and problem formulation

LOTUS is framed around a recurrent production problem in short-form video editing: creators repurpose long-form videos into much shorter outputs, but must simultaneously identify salient moments, preserve or reconstruct a coherent story, compress content to social-media length, maintain strong hooks and endings, avoid awkward cuts, and sometimes decouple audio and visuals so narration remains concise while visuals stay interesting [2502.07096]. In genres such as tours, travel, reviews, and tutorials, important information is often distributed across the source video and conveyed jointly through narration and visuals. Editors in the formative study described this process as taking hours to days, largely because of clip selection and sequencing [2502.07096].

The system distinguishes two modes of short-form generation. **Extractive** short videos are composed of clips directly taken from the long-form source. **Abstractive** short videos introduce newly generated narration and then pair that narration with visuals drawn from the source video. LOTUS does not treat these as mutually exclusive alternatives. Instead, it treats them as complementary mechanisms whose relative advantages depend on the structure of the source material and the desired editorial effect [2502.07096].

This design leads to a specific interpretation of short-form creation: not as a single summarization problem, but as a coordination problem across script compression, shot selection, audiovisual alignment, and authenticity management. A plausible implication is that LOTUS should be understood less as a conventional summarizer than as a hybrid authoring system in which automated summarization and editorial control are deliberately interleaved.

## 2. Abstractive summary video generation

The first stage of LOTUS creates an **abstractive summary video** from the long-form input [2502.07096]. Starting from the long-form transcript, LOTUS prompts GPT-4o to generate a short-form transcript in the style of the original speaker, explicitly encouraging a strong social-media-friendly hook and reuse of the original introduction and conclusion. To better preserve visual coverage, LOTUS extracts noun phrases from the long-form transcript using spaCy and includes them in the prompt as candidate visuals [2502.07096].

After generating the short-form transcript, LOTUS identifies **visual concepts**: concrete words or phrases in the new script that should correspond to shots. This is done by scene-detecting the long video into clips, generating clip descriptions with GPT-4o, and prompting GPT-4o to embed suitable visual concepts directly into the short-form sentences [2502.07096]. The source video is therefore not matched to narration only after summarization; visual anchoring is introduced into the script generation process itself.

Clip selection is then handled as a weighted matching problem. Each long-form clip is scored against a target visual concept using four signals:

1. **Speech similarity** between the clip transcript and the visual concept, using `nomic-embed-text` embeddings.  
2. **Keyframe similarity** between the clip’s keyframe and the visual concept, using CLIP.  
3. **GPT-4o scoring**, where GPT-4o rates a filtered set of about 25 candidate clips given their keyframes and speech.  
4. **Position-based alignment**, which rewards clips appearing at roughly the same relative point in the source video as the concept appears in the generated summary [2502.07096].

The position score is given as

\[
Score_{pos} = 1 - Pos(visual\ concept) - Pos(long-form\ clip)
\]

where `Pos(visual concept)` is the concept’s normalized position in the short-form transcript and `Pos(long-form clip)` is the candidate clip’s normalized position in the long-form video [2502.07096]. After scoring, LOTUS assigns the highest-scoring long-form clip to each visual concept. If the same source clip is selected multiple times, the system iteratively replaces duplicates with the next-best alternatives until duplicates are removed [2502.07096].

The script is then synthesized into narration using ElevenLabs, and the matched clips are aligned to the generated speech. LOTUS adjusts clip playback speed when necessary so clip duration fits narration duration, producing the initial abstractive short video shown to the user [2502.07096].

This stage is notable because the abstraction is script-driven rather than clip-driven. The short-form video is first defined as a compressed narrative and only then grounded in source visuals. This suggests that LOTUS treats temporal structure and rhetorical framing—especially the hook and conclusion—as primary organizational units, with clip retrieval serving that structure rather than determining it.

## 3. Blended abstractive-extractive construction

The second stage produces a **blended abstractive-extractive video** by selectively replacing some generated-narration segments with extractive segments from the original video [2502.07096]. The rationale is that certain moments benefit from preserving the original creator’s voice, camera-facing delivery, or naturally synchronized audiovisual pairing.

To do this, LOTUS segments both the generated abstractive script and the long-form transcript. The abstractive script is segmented with GPT-4o; the long-form transcript is segmented using ROPE’s segmentation algorithm [2502.07096]. It then compares every abstractive segment to extractive candidates using four metrics:

1. **Speech similarity** between abstractive and extractive segments.  
2. **Visual connection**, defined as the similarity between a segment’s speech and its keyframe via CLIP.  
3. **Coverage**, the number of noun phrases present in the segment speech, extracted by spaCy.  
4. **Position-based alignment**, based on relative positions of abstractive and extractive segments [2502.07096].

If the difference between an abstractive segment and its highest-scoring extractive alternative is below an empirically chosen threshold, LOTUS keeps both as plausible options and generates permutations of the video using those choices. Among these candidate mixtures, it selects the one with the highest coherence score, using **GPT-3 loss** as the coherence criterion [2502.07096]. The selected sequence becomes the final blended short-form video. LOTUS then applies **volume normalization** and **denoising** so audio sounds consistent across generated and extracted segments [2502.07096].

The resulting end-to-end workflow is:

input long-form video → scene detection and transcript extraction → noun phrase extraction → GPT-4o short-form transcript generation → visual concept extraction → scoring and alignment of long-form clips to summary concepts → generated narration + clip timing adjustment to form initial abstractive result → segmentation of both summary and source transcripts → candidate extractive replacement scoring → permutation search and coherence optimization → final blended short video [2502.07096].

This architecture makes an important distinction between two kinds of fidelity. One is **content fidelity**, preserved by the script-driven abstractive summary and source-clip matching. The other is **performative fidelity**, restored when extractive segments preserve the original creator’s voice and audiovisual timing. LOTUS operationalizes both.

## 4. Interaction model and editing interface

The interaction model is central to LOTUS. After the automated pipeline produces a draft, the user edits and refines it through an interface with three panes: a video player pane, a long-form clips pane, and a short-form clips pane [2502.07096]. The long-form pane presents source clips in grid or list form, with keyframes and optionally speech, allowing users to skim the source at clip granularity rather than scrubbing a full timeline. It also supports **CLIP-based search** by text prompt or by selecting a keyframe, reordering clips by similarity [2502.07096].

The short-form pane shows the current summary clips and supports direct manipulation. Creators can refine the generated result in several ways. They can **toggle any clip between abstractive and extractive modes**. In abstractive mode, they can edit narration text, generate alternative text, and choose a speaker voice from diarized speakers found via **PyAnnote** or from a default voice. In extractive mode, they can preserve source audio and optionally denoise it [2502.07096].

The editing operations include dragging source clips into the short-form sequence, reordering clips, deleting clips, trimming or extending clips, and replacing a clip with one of 20 suggested **alternative clips** that better match the same transcript section [2502.07096]. LOTUS also provides an **align** feature that maps a short-form clip back to its location in the long-form clip list and shows nearby context, enabling users to pull in surrounding clips or borrow wording from adjacent transcript sections [2502.07096].

The system’s design choices follow three stated goals from the formative work: help users identify key clips, help them arrange clips into coherent sequences, and let them flexibly decouple and recombine audio and visuals [2502.07096]. This led to several concrete interface decisions: beginning from a generated short-form structure rather than a blank long video, representing the source as clips with keyframes and captions, allowing local rewriting of narration, exposing search and alternative clip suggestions, and preserving opportunities to use the original creator’s voice for authenticity [2502.07096].

The implementation uses a React frontend, Flask backend, and a separate Flask API for logging; video and audio processing use MoviePy and torchaudio [2502.07096].

## 5. Prompting, segmentation, and algorithmic control surfaces

Although LOTUS is presented as a video editing system, much of its algorithmic behavior is determined by prompt design rather than by closed-form optimization objectives. The paper explicitly lists several important prompts: one for short-form transcript generation in TikTok/Instagram/Shorts style, one for visual concept extraction that embeds `<VIS> ... </VIS>` spans into sentences, one for multimodal GPT-4o clip scoring from 0 to 1, one for transcript segmentation into `<TITLE>` and `<SEG>` blocks, and one for generating alternative speech for a selected shot given before/after context and shot duration [2502.07096].

These prompts are not incidental interface features. They define the summary script, visual anchors, segmentation structure, clip scores, and editable rewrites. In that sense, LOTUS uses LLM prompting as a control mechanism inside the summarization pipeline itself [2502.07096].

This has two consequences. First, the system’s behavior is partly contingent on language-model instruction following rather than solely on deterministic ranking heuristics. Second, the editable draft becomes an important stabilizing layer: creators can revise generated text, swap in extractive material, and adjust clip structure after the LLM-defined initial pass. A plausible implication is that the interface is not merely post-processing; it is part of the reliability strategy.

## 6. Evaluation, genre effects, and authorship implications

The evaluation has two parts: a **results evaluation** comparing generated videos, and a **user study** comparing editing workflows [2502.07096].

In the results evaluation, the authors compared three methods: an extractive baseline adapted from ROPE, LOTUS’s **abstractive** method, and LOTUS’s **blended abstractive-extractive** method [2502.07096]. They selected six YouTube videos from trending content, each 5–20 minutes long and visually diverse, excluding podcasts, TED talks, and lectures. The videos covered travel, cooking, spotlight/interview, and home tour content, with source durations from 7:25 to 17:18 and generated short versions roughly 50 seconds to 1:25 [2502.07096]. Twelve annotators familiar with short-form video ranked outputs by overall preference and rated hook, narrative, and visual appeal on 5-point Likert scales [2502.07096].

The blended method had the best average overall ranking:

- mixed: \(p = 1.88, \sigma = 0.72\)  
- abstractive: \(p = 2.06, \sigma = 0.77\)  
- extractive: \(p = 2.04, \sigma = 0.82\)

but the difference was not statistically significant [2502.07096]. More informative than the aggregate ranking was the genre pattern. Extractive and mixed outputs were preferred for videos where a narrator was prominently on camera, because preserving the source speaker increased authenticity and engagement. Abstractive outputs were preferred when the source did not strongly depend on a visible narrator, because generated narration made it easier to cover more content concisely [2502.07096]. Missing hooks and conclusions were especially damaging for extractive methods; in some cases the mixed method prevailed because it combined an abstractive storyline with an extracted opening hook [2502.07096].

The user study used a within-subjects design with 8 participants who had video editing experience, recruited via Upwork and university mailing lists. Participants averaged 22 years old and reported 3.71 years of editing experience on average [2502.07096]. They used both LOTUS and their preferred existing editing tool, such as Adobe Premiere Pro or CapCut, on two selected long-form YouTube videos of similar length and visual diversity, with up to 20 minutes per task after 10 minutes of familiarization per source video [2502.07096].

The findings were positive for LOTUS as an authoring aid. All participants said they would want to use it in the future. They produced short-form videos they considered comparable in quality to those made with their usual tools, without increased mental demand, and generally liked the editing process in LOTUS more [2502.07096]. Several participants valued the fact that LOTUS starts with a coherent draft rather than requiring them to cut down from the full source. The initial result was described as especially useful as a structure or “base to go off of” [2502.07096]. Common interactions included align, drag shot, delete shot, trim shot, and render, along with search, text editing, and alternative clip replacement [2502.07096].

The qualitative findings also clarify how creators divided labor between clip types. **Abstractive clips** were valued for flexibility, because users could say exactly what they wanted, shorten long narration, and pair preferred visuals with revised speech. **Extractive clips** were valued for credibility, personality, and naturalness, because the original creator’s voice and planned audiovisual delivery felt more authentic [2502.07096].

These results complicate any simple claim that one summarization mode is superior. LOTUS’s empirical contribution is not that hybridity always wins numerically, but that hybridization exposes a controllable tradeoff between compression and authenticity that varies by genre and presentation style.

## 7. Limitations and future directions

The paper identifies several limitations. Generated speech sometimes sounded unnatural or lacked expressive nuance, especially with multiple speakers, reducing the appeal of abstractive clips [2502.07096]. Performance also depends on video type: instructional videos are challenging because they require precise step-by-step audiovisual alignment, and existing source visuals may not adequately cover a concise rewritten script [2502.07096].

Participants additionally noted interface limitations relative to mature editors: no traditional horizontal timeline, limited trimming granularity, latency before hearing regenerated speech, and fewer low-level controls such as independently editing audio and video channels or adjusting aspect ratio [2502.07096]. These observations matter because LOTUS is presented not as a replacement for full-featured editing software, but as a hybrid human-AI video editing system [2502.07096].

The authors suggest future work on better hooks and conclusions, B-roll recommendation or generation, more expressive speech synthesis, natural-language editing commands, multiple starting drafts, and integration with existing editing tools [2502.07096]. This suggests an ongoing shift from summarization-as-output toward summarization-as-editable-structure.

A common misconception would be to treat LOTUS as a fully automatic short-video generator. The paper explicitly positions it otherwise: its core contribution is showing that short-form creation benefits from starting with an **abstractive, script-driven summary** for compression and coverage, then selectively recovering **extractive source clips** to restore authenticity, natural audiovisual alignment, and engaging moments [2502.07096]. In that formulation, automation provides a strong draft, but authorship remains distributed across generation, ranking, and human refinement.

Source: https://www.emergentmind.com/topics/lotus