Controllable Hybrid Captioner

• The paper introduces a unified hybrid captioner that generates complementary action and scene descriptions, enriching video memory for effective QA.
• It employs explicit control tokens ([ACX] for actions and [SCX] for scenes) to switch between dynamic event descriptions and static scene layouts.
• Empirical results show improved semantic similarity and QA accuracy, demonstrating the method’s strength in long-form egocentric video understanding.

Searching arXiv for the named paper and closely related controllable captioning work to ground the article with current references. A Controllable Hybrid Captioner is a video captioning framework that generates two complementary kinds of descriptions—short action/event captions and static scene descriptions—and can be explicitly instructed which type to emit via control tokens (Sasse et al., 22 Jul 2025). In the long-form video understanding setting addressed by recent work, the framework is used to build a progressive text-based memory from shorter video chunks, with the resulting activity log supplied to a LLM for question answering over multi-minute egocentric video (Sasse et al., 22 Jul 2025). More broadly, the concept sits within controllable captioning research that conditions generation on explicit control signals such as sentence length, sentence tense, number of nouns, region sequences, style prompts, or multimodal user instructions (Zhu et al., 2021, Wang et al., 2022, Wang et al., 2023). In this narrower sense, the “hybrid” designation refers specifically to the alternation between action captions and scene captions under explicit control, while a broader interpretation connects it to architectures that combine grounding, structure, style, and modality-specific control in a unified captioning pipeline (Sasse et al., 22 Jul 2025, Wang et al., 2023).

1. Definition and scope

In the long-form video setting, the Controllable Hybrid Captioner is introduced as a unified captioner that can generate action captions and scene captions and can alternate between them according to special input tokens that signal scene changes detected in the video (Sasse et al., 22 Jul 2025). The framework is motivated by the observation that caption logs built solely from short video captions are mostly action-centric, whereas downstream questions may depend on scene layout, visible objects, object attributes, relations among objects, and other contextual cues that persist across time (Sasse et al., 22 Jul 2025). The hybrid captioning strategy therefore enriches the activity log with static scene descriptions in addition to action descriptions (Sasse et al., 22 Jul 2025).

This places the framework within a larger research program on controllable captioning. Existing image captioning models have been extended with control signals over macroscopic sentence attributes, including sentence quality, sentence length, sentence tense and number of nouns (Zhu et al., 2021). Other work studies structure-related control signals such as sentence length and Part-of-Speech tag sequences (Zhu et al., 2021), prompt-based switching among domains and styles such as COCO-style, TextCaps-style, short-length, medium-length, high-length, positive, and negative captions (Wang et al., 2022), or ordered region-set control for grounded descriptions (Cornia et al., 2018). The Controllable Hybrid Captioner differs from these by focusing on caption-type control in long-form video memory construction, but it shares the same basic principle: explicit conditional control over the form or content of the generated description (Sasse et al., 22 Jul 2025).

A plausible implication is that the framework can be regarded as a domain-specific instantiation of controllable captioning rather than a wholly separate paradigm. Its distinctive contribution lies in using control not merely to vary style or syntax, but to decide whether the memory should record dynamic activity or static scene context at a given point in a long video (Sasse et al., 22 Jul 2025).

2. Long-form video memory construction

The framework is developed for long-form video understanding, especially multi-minute egocentric videos such as those in EgoSchema (Sasse et al., 22 Jul 2025). The system follows a progressive memory construction approach: a captioner operates on short video chunks where spatio-temporal modeling is computationally feasible, the outputs are appended into a running text log, and the final log is provided to an LLM for question answering (Sasse et al., 22 Jul 2025). The paper explicitly states that, unlike LLoVi, it does not use multi-round prompting or query-based summarization, so improvements are attributed to better memory construction rather than elaborate prompting (Sasse et al., 22 Jul 2025).

The overall pipeline has four main stages: partition the long video into shorter units, generate captions over those units, accumulate those captions into a text memory or activity log, and feed the memory to an LLM for question answering (Sasse et al., 22 Jul 2025). This architecture reflects the view that raw video is a poor interface for downstream reasoning, whereas a text-based memory is much more compact than raw frames, naturally consumable by an LLM, easier to accumulate progressively over time, and better suited for question answering that requires integrating evidence across distant moments (Sasse et al., 22 Jul 2025).

Within this pipeline, the hybrid captioning idea addresses a specific deficiency of action-only logs. A short action caption can omit contextual details that matter for downstream reasoning, such as whether a bottle is on a kitchen counter, next to a cutting board, beside a sink, or whether the scene is outdoors versus indoors (Sasse et al., 22 Jul 2025). By interleaving scene descriptions with action descriptions, the resulting log becomes more detailed and complete, and the space of answerable questions expands (Sasse et al., 22 Jul 2025).

This suggests that the Controllable Hybrid Captioner is best understood not as a standalone captioning endpoint but as a memory-building component in a larger reasoning system. Its captions are designed to be concise, temporally ordered, and directly useful for later language-based inference (Sasse et al., 22 Jul 2025).

3. Action–scene hybridization and control tokens

The framework is hybrid because it produces two distinct caption modalities: action captions and scene captions (Sasse et al., 22 Jul 2025). Action captions are short clip-level event descriptions produced by LaViLa and are characterized in the paper as focused on human actions (Sasse et al., 22 Jul 2025). Scene captions are static descriptions of the visual environment, centered on objects, properties, and relations, and in the ensemble setting LLaVA is prompted to “Describe the scene as specifically as possible focusing on objects and their properties and their relations to other objects in the scene. Be as concise as possible like you are writing a log.” (Sasse et al., 22 Jul 2025)

It is controllable because the desired caption type is specified through special tokens added to LaViLa’s GPT-2 decoder vocabulary: $[ACX]$ for action caption and $[SCX]$ for scene caption (Sasse et al., 22 Jul 2025). During training, ground-truth action captions are paired with $[ACX]$ and synthetic scene captions are paired with $[SCX]$; at inference, the same tokens are used to trigger the target caption mode (Sasse et al., 22 Jul 2025). The intended operating mode is to request action captions on regular short video chunks and scene captions when a scene change detector fires (Sasse et al., 22 Jul 2025).

This token-based conditioning closely parallels other controllable captioning mechanisms. “Show, Control and Tell” represents region-pointer advancement as an explicit generated [NEXT](https://www.emergentmind.com/topics/neural-external-torque-estimation-next) token inside the decoder, turning a control-flow decision into part of the sequence model (Lindh et al., 2020). “Learning Combinatorial Prompts for Universal Controllable Image Captioning” represents heterogeneous control signals as prompt components concatenated into a unified prompt sequence for GPT-2 (Wang et al., 2023). “Controllable Image Captioning via Prompting” uses manual or learned prompts to switch between domains and styles in a unified BLIP-based captioner (Wang et al., 2022). The Controllable Hybrid Captioner adopts the same general logic of discrete control symbols, but its symbols control caption type rather than region advancement or style (Sasse et al., 22 Jul 2025).

A plausible implication is that caption-type control via $[ACX]$ and $[SCX]$ acts as a low-dimensional interface between memory policy and caption realization. Instead of asking the decoder to infer when to describe actions or scenes implicitly, the policy is externalized through explicit control tokens (Sasse et al., 22 Jul 2025).

4. System architecture and components

The system uses several distinct components (Sasse et al., 22 Jul 2025):

Component	Role
LaViLa	Base short-video captioner for action captions
LLaVA	Vision-LLM for static scene descriptions
LaViLa-CHC	Unified controllable hybrid captioner
Scene segmentation module	Detects scene changes
Llama-3.1-70B-Instruct	Final QA model over the text memory

In the initial ensemble pipeline, LaViLa produces action captions over short video clips, and when the segmentation module detects a scene change, LLaVA is prompted on a representative frame to produce a concise scene description (Sasse et al., 22 Jul 2025). Both outputs are appended into a single chronological log (Sasse et al., 22 Jul 2025). The main contribution is then the unified Controllable Hybrid Captioner, LaViLa-CHC, which replaces the separate scene-caption model by fine-tuning LaViLa so the same captioner can emit either action or scene captions depending on the prepended control token (Sasse et al., 22 Jul 2025).

The segmentation module determines when scene changes occur, and the caption type is selected by prepending the appropriate token to the decoder input (Sasse et al., 22 Jul 2025). The caption log thus alternates between dynamic and static entries only when needed, rather than redundantly generating a scene description at every step (Sasse et al., 22 Jul 2025). The final activity log is chronological and progressive, and scene descriptions are inserted only when the system believes the scene has changed (Sasse et al., 22 Jul 2025).

This architecture has clear affinities with other hybrid captioning designs. “Caption Anything” formalizes controllable captioning as a modular triplet of segmenter, captioner, and text refiner, supporting both visual controls and language controls through sequential modular composition (Wang et al., 2023). “Controllable Contextualized Image Captioning” combines image content, textual context, and highlighted context spans through either prefix prompting or encoder recalibration (Mao et al., 2024). “AnyCapModel” refines captions from frozen base captioners using modality features and user instructions in a plug-and-play framework for image, video, and audio (Ren et al., 17 Jul 2025). In comparison, the Controllable Hybrid Captioner is narrower in scope but similarly modular: segmentation decides when the control should switch, captioning generates memory entries, and a downstream LLM reasons over the result (Sasse et al., 22 Jul 2025).

5. Segmentation, training, and decoding

Segmentation is central because scene captions are inserted only at detected scene changes (Sasse et al., 22 Jul 2025). The paper explores three strategies: Uniform segmentation, PySceneDetect, and Kernel Temporal Segmentation (KTS) (Sasse et al., 22 Jul 2025). Uniform segmentation assumes scene changes occur uniformly every 120 seconds (Sasse et al., 22 Jul 2025). PySceneDetect uses the ContentDetector algorithm, measuring changes in color and intensity between adjacent frames and comparing a weighted average of pixel changes against a threshold (Sasse et al., 22 Jul 2025). KTS is described as combining spatio-temporal video segmentation, region tracking, multiple cues such as color, edges, motions, and kernel-based models, using ViT-B/32 CLIP features from sampled frames in the implementation (Sasse et al., 22 Jul 2025).

The paper reports that PySceneDetect detected about 30 scenes on average and KTS detected about 20 scenes on average (Sasse et al., 22 Jul 2025). The best strategy depends on the pipeline: for the LaViLa + LLaVA ensemble, KTS gave the best QA result, whereas for LaViLa-CHC, uniform segmentation gave the best result (Sasse et al., 22 Jul 2025).

Training LaViLa-CHC requires both action-caption and scene-caption supervision (Sasse et al., 22 Jul 2025). The base is LaViLa, whose language decoder is GPT-2-medium with 137M parameters (Sasse et al., 22 Jul 2025). To create scene-caption supervision, the authors randomly sample 350 Ego4D videos, exclude EgoSchema videos to avoid leakage, select the center frame among 32 uniformly sampled frames from each short clip, and run LLaVA on that frame to produce a scene description, yielding about 200k scene-caption examples (Sasse et al., 22 Jul 2025). Ground-truth action captions are paired with $[ACX]$, synthetic scene captions with $[SCX]$, and the model is fine-tuned jointly on both (Sasse et al., 22 Jul 2025).

A practical issue arose because GPT-2 is a plain causal LLM without instruction tuning or RLHF; when moved from single-sentence action captions to paragraph-long scene descriptions, the model sometimes generated repetitive text (Sasse et al., 22 Jul 2025). To address this, the authors applied a repetition penalty following CTRL, testing values 1, 1.2, 1.5, 2, and 3, and found 3 worked best for most setups based on QA performance (Sasse et al., 22 Jul 2025). The training objective is not written as an explicit formula in the paper, but standard autoregressive caption generation conditioned on visual features and the control token is implied (Sasse et al., 22 Jul 2025).

This training regime has a clear analogue in prompt-based and token-based controllable captioning. Prompt-conditioned unified captioners learn multiple domains or styles without changing the base decoding objective (Wang et al., 2022), while token-conditioned grounded captioners incorporate control operations directly into the decoder vocabulary (Lindh et al., 2020). The Controllable Hybrid Captioner adopts the token-conditioned route and combines it with synthetic teacher-generated supervision (Sasse et al., 22 Jul 2025).

6. Empirical results and broader significance

The main downstream benchmark is EgoSchema, described as containing over 5,000 multiple-choice QA pairs, over 250 hours of video, three-minute clips, and five answer options per question (Sasse et al., 22 Jul 2025). The baseline action-only system, LaViLa/LLoVi, achieves 41.4% multiple-choice QA accuracy (Sasse et al., 22 Jul 2025). In the ensemble setting, adding scene captions improves results substantially, with KTS + LLaVA 34B reaching 57.2 (Sasse et al., 22 Jul 2025). In the unified CHC setting, the strongest result is 52.4% with uniform segmentation, scene captions enabled, and a 34B teacher (Sasse et al., 22 Jul 2025). The paper also reports that CHC performed better than the ensemble LaViLa+LLaVA on 7B setup while requiring a much smaller memory (Sasse et al., 22 Jul 2025).

Caption similarity results further characterize the model (Sasse et al., 22 Jul 2025). For action captions, LaViLa-CHC improves SentenceBERT semantic similarity from 0.49 for the LaViLa baseline to 0.66 for both 7B- and 34B-teacher variants, while BLEU remains 13.71 (Sasse et al., 22 Jul 2025). For scene captions, lexical metrics are low, but SentenceBERT reaches 0.63 with a 7B teacher and 0.68 with a 34B teacher (Sasse et al., 22 Jul 2025). The authors attribute low lexical overlap partly to the longer, more varied phrasing of scene captions (Sasse et al., 22 Jul 2025).

The framework’s significance extends beyond its immediate benchmark. It shows that conditional generation over caption type can improve long-form video memory construction without requiring separate deployed captioning models for dynamic and static content (Sasse et al., 22 Jul 2025). This is conceptually aligned with broader controllable captioning findings: length-aware models can generate captions of different descriptiveness levels (Luo et al., 2020); prompt-conditioned captioners can switch styles or domains at test time by changing only the prompt (Wang et al., 2022); combinatorial prompt methods can handle multiple control signals jointly by learning prompt blocks for different signal types (Wang et al., 2023); and post-hoc refiners can enhance controllability without retraining frozen base models (Ren et al., 17 Jul 2025).

At the same time, the paper makes several limitations explicit. The whole QA pipeline depends on caption quality; scene-change detection is imperfect; scene captions are learned from synthetic LLaVA outputs rather than human annotations; repetition required inference-time penalty tuning; and there is a tradeoff between memory length and informativeness (Sasse et al., 22 Jul 2025). The conclusion suggests extending controllable generation to other information sources, such as object detection outputs and semantic segmentation outputs, and exploring smaller QA LLMs under the hypothesis that a richer caption log may reduce the need for very large reasoning models (Sasse et al., 22 Jul 2025).

Taken together, these results position the Controllable Hybrid Captioner as a specific but influential design pattern within controllable captioning: a control-token-conditioned, jointly trained, multi-caption-type captioner that enriches textual memory by alternating dynamic and static descriptions under explicit control (Sasse et al., 22 Jul 2025). In the broader landscape of controllable captioning, it exemplifies a recurring theme: hybrid systems become more useful when control is made explicit, modular, and directly tied to the requirements of downstream reasoning (Sasse et al., 22 Jul 2025, Wang et al., 2023, Wang et al., 2023, Ren et al., 17 Jul 2025).