CapTune: Adaptive NSI Caption Personalization

• CapTune is a system for adapting non-speech captions, enabling deaf and hard of hearing viewers to personalize content within creator-defined limits.
• The system employs an anchored generative model, allowing adjustments along levels of detail and expressiveness while preserving original intent.
• CapTune integrates UI constraints, context grounding, and human oversight to deliver safe, consistent, and emotionally engaging caption adaptations.

Searching arXiv for CapTune and closely related papers to ground the article. CapTune is a system for customizing non-speech captions for deaf and hard of hearing (DHH) viewers while preserving creator intent. In the formulation introduced in "CapTune: Adapting Non-Speech Captions With Anchored Generative Models" (Huang et al., 27 Aug 2025), the target object is non-speech information (NSI) in timed text, including sound effects, music cues, environmental audio, and paralinguistic information such as tone, intensity, rate, and sarcasm markers. Rather than treating NSI as a single static caption track, CapTune frames caption semantics as a co-authored, anchored generative space: caption creators define acceptable transformation bounds, and viewers personalize captions within those bounds across level of detail, expressiveness, sound representation method, and genre alignment.

1. Problem Setting and Scope

CapTune addresses a recurring limitation in caption authoring practice: conventional workflows provide one caption track even though DHH viewers have heterogeneous preferences and viewing contexts (Huang et al., 27 Aug 2025). Existing standards and style guides emphasize timing, legibility, and minimal NSI conventions, but the system is motivated by the observation that such conventions still impose “one-size-fits-all” semantics. The central design problem is therefore not simply whether NSI should be present, but how much information it should carry, how interpretive it should be, how sounds should be represented textually, and how strongly captions should align with genre tone.

Within this framing, non-speech captions are treated as semantically transformable rather than merely format-constrained. CapTune is not an open-ended automatic caption generator. Its defining premise is that personalization should remain subordinate to creator-approved bounds. This is why the system is described as an “anchored generative” approach rather than unconstrained text synthesis: creators specify the safe region, and viewers navigate only within that region (Huang et al., 27 Aug 2025).

A common misunderstanding is to equate caption personalization with arbitrary stylistic rewriting. CapTune is explicitly designed against that interpretation. The paper formalizes personalization as bounded adaptation under creator control, with UI constraints, anchors, context grounding, and item locking all functioning as safeguards. This suggests that the system’s contribution lies as much in constraining generation as in enabling it.

2. Anchored Generative Model and Safe Transformation Space

The core abstraction in CapTune is the “safe transformation space” defined by creator-specified anchors along two axes: Level of Detail and Expressiveness (Huang et al., 27 Aug 2025). For each caption item, the lower anchor denotes the minimally acceptable detail or expressiveness that still preserves clarity, and the upper anchor denotes the richest or most expressive formulation the creator endorses. Viewer-side generation is then constrained to the creator-defined rectangular region in this two-dimensional space.

The system formalizes the relation between a viewer’s UI input and the semantic target value with a piecewise linear mapping centered on the original caption, which serves as the baseline at $s_0 = 0$:

$f(s)= \begin{cases} V_{\min} + \dfrac{s-s_{\min}}{s_0-s_{\min}} (V_0 - V_{\min}), & s < s_0 \[0.6em] V_0 + \dfrac{s-s_0}{s_{\max}-s_0} (V_{\max} - V_0), & s > s_0 \end{cases}$

The typical ranges are $[s_{\min}, s_{\max}] = [-10, 10]$ and $[V_{\min}, V_{\max}] = [1, 10]$ (Huang et al., 27 Aug 2025). This construction decouples UI position from semantic value and allows the original caption to remain the reference point even when creator-imposed bounds are asymmetric.

At inference time, the viewer request is quantified relative to the anchor box using the ratios

$$r = \frac{V' - V_{\min}}{V_{\max} - V_{\min}}, \qquad r \in [0,1]$$

and

$$\delta = \frac{V' - V}{V_{\max} - V_{\min}}, \qquad \delta \in [-1,1].$$

These values are inserted into a structured prompt for GPT-4o, together with the original caption, the lower and upper anchor exemplars, and scene context (Huang et al., 27 Aug 2025). The purpose is not to ask the model to invent a caption de novo, but to interpolate toward a creator-approved target style. The paper further notes a latent coupling between detail and expressiveness: increasing one can inadvertently shift the other. CapTune therefore recalibrates the untouched parameter after a preview is generated so that the UI remains semantically consistent.

This architecture establishes an important conceptual distinction. The system does not define safety through an explicit post-hoc semantic distance threshold, even though the paper notes that such a constraint could in principle be written as $d(y, y_{\text{anchor}}) \le \epsilon$. Instead, safety is implemented through hard UI bounds, anchored prompts, context grounding, and human oversight (Huang et al., 27 Aug 2025).

3. Personalization Dimensions

CapTune operationalizes four personalization dimensions derived from DHH community analysis. Two of them—Level of Detail and Expressiveness—are creator-bounded continuous parameters. The other two—Sound Representation Method and Genre Alignment—are viewer-controlled preferences applied within the permitted detail/expressiveness region (Huang et al., 27 Aug 2025).

Dimension	Definition	Reported effect
Level of detail	How much descriptive content is included	Higher detail can improve narrative clarity but increases reading load
Expressiveness	From neutral to evocative phrasing	More expressive captions can heighten emotional engagement but may feel intrusive
Sound representation method	Source-focused, onomatopoeia, or sensory quality-focused	Different modes trade off disambiguation, vividness, and accessibility
Genre alignment	Align caption tone with genre or mood	Can strengthen thematic coherence and immersion

The paper illustrates these dimensions with concrete caption transformations. For level of detail, a caption such as “[Music playing]” can become “[Soft piano melody with rising tempo]”. For expressiveness, “[Door closes]” can become “[Door slams shut with a resonant thud]”. For genre alignment, “[Wind blowing]” can shift toward “[Whimsical swoosh]” in animation, “[Freezing wind]” in dark fantasy, or “[Industrial hum]” in science fiction (Huang et al., 27 Aug 2025). These examples are not merely stylistic ornamentation; they expose the design tension between information density, interpretive language, and cognitive burden.

The sound representation dimension is especially significant because it separates several textual functions that conventional NSI often collapses. Source-focused captions emphasize what produced the sound. Onomatopoeia attempts to mimic the sound. Sensory quality-focused captions describe pitch, texture, or intensity. The evaluation reports that source labels aid disambiguation, onomatopoeia can be vivid but also ambiguous or inaccessible for some Deaf viewers, and sensory descriptors can help communicate abstract or high-frequency sounds without excessive interpretation (Huang et al., 27 Aug 2025).

A plausible implication is that CapTune treats caption adaptation as a multidimensional semantic control problem rather than a single scalar notion of “more descriptive” or “less descriptive.” That framing differentiates it from presentation-oriented accessibility tools that manipulate typography or placement without altering NSI semantics.

4. System Architecture and Workflow

CapTune comprises two principal components: a Creator Tool and a Viewer Client (Huang et al., 27 Aug 2025). The Creator Tool accepts a video file and caption file in .srt format. It highlights bracketed or parenthetical captions as NSI candidates, extracts a 10-second clip around each candidate, and analyzes that clip with VideoLLaMA2.1-7B-AV to produce textual scene descriptors. GPT-4o then assigns baseline Level of Detail and Expressiveness values on a 1–10 scale to the original caption. Creators manipulate sliders to set lower and upper anchors, and each change triggers a GPT-4o preview grounded in the original caption and scene context. If necessary, the creator can manually edit the preview or lock the item. The export is a configuration JSON containing original captions, per-item anchor bounds, calibrated baseline values, scene context, and video metadata such as title and genre.

The Viewer Client exposes the personalization interface. Its central control is a 10×10 style grid with Level of Detail on the x-axis and Expressiveness on the y-axis, with only creator-permitted cells enabled. The original caption cell is selected by default. The interface also exposes sound representation and genre alignment controls, plus a natural-language interface through which viewers can state goals such as keeping captions brief while clarifying sound sources (Huang et al., 27 Aug 2025). The client then computes the interpolation ratios $r$ and $\delta$, constructs a structured prompt, sends it to GPT-4o, and renders the returned caption in sync with playback.

The implementation stack is heterogeneous but straightforward: React and TypeScript on the front end; Node.js on the back end for caption parsing, preference management, and GPT-4o calls; and a Python module running VideoLLaMA2 analysis on a remote server. Deployment is reported on Windows 11 Enterprise with an RTX 4090 GPU. The source code is open at GitHub.com/SoundabilityLab/CapTune, and the system relies on prompting rather than custom training or finetuning (Huang et al., 27 Aug 2025).

The guardrails are distributed across the pipeline. The style grid disables cells outside $$[\mathrm{LoD}_{\min}, \mathrm{LoD}_{\max}] \times [\mathrm{Expr}_{\min}, \mathrm{Expr}_{\max}]$$. Locked items bypass transformation. The prompt always includes the original caption and anchor exemplars. Scene descriptors from VideoLLaMA2 reduce hallucination or misinterpretation. Creators can manually edit or lock outputs when nuance is critical. In this sense, CapTune is best understood as a human-in-the-loop constrained generation system rather than a fully autonomous captioning model.

5. Evaluation and Reported Findings

The system was evaluated through a Creator Tool study with seven caption creators and a Viewer Client study with twelve DHH participants (Huang et al., 27 Aug 2025). The creator study used remote 60-minute sessions with a concept overview, a walkthrough of the tool, guided tasks on three clips, think-aloud protocol, and a semi-structured interview. Analysis used open, axial, and selective coding, with 72 open codes consolidated into 8 themes. The viewer study used remote sessions of approximately 90 minutes, included interpreters or CART as needed, and analyzed transcripts with a three-level codebook. Reported interrater reliability was Cohen’s $f(s)= \begin{cases} V_{\min} + \dfrac{s-s_{\min}}{s_0-s_{\min}} (V_0 - V_{\min}), & s < s_0 \[0.6em] V_0 + \dfrac{s-s_0}{s_{\max}-s_0} (V_{\max} - V_0), & s > s_0 \end{cases}$0 with raw agreement of 90.5%.

Across the creator study, participants understood the two-anchor metaphor for defining a two-dimensional transformation space, although some initially proposed four anchors before concluding that two were sufficient (Huang et al., 27 Aug 2025). Creators valued the edit-and-lock safeguards and the balance between accessibility and intent. They also emphasized contextual dependence: acceptable settings varied with genre, pacing, and communication goals, and some requested scene-level configurations. Concrete concerns centered on over-specific interpretations and inaccurate sensory descriptions, such as describing a sound as “cracking” rather than “freezing.”

The viewer study reports that most participants, specifically $f(s)= \begin{cases} V_{\min} + \dfrac{s-s_{\min}}{s_0-s_{\min}} (V_0 - V_{\min}), & s < s_0 \[0.6em] V_0 + \dfrac{s-s_0}{s_{\max}-s_0} (V_{\max} - V_0), & s > s_0 \end{cases}$1, felt customization deepened emotional connection with the content (Huang et al., 27 Aug 2025). At the same time, many participants, specifically $f(s)= \begin{cases} V_{\min} + \dfrac{s-s_{\min}}{s_0-s_{\min}} (V_0 - V_{\min}), & s < s_0 \[0.6em] V_0 + \dfrac{s-s_0}{s_{\max}-s_0} (V_{\max} - V_0), & s > s_0 \end{cases}$2, described the tuning process as trial-and-error that competed with attention to the video itself. The 10×10 grid was often perceived as overly granular, and participants requested previews, comparison toggles, and saved profiles or presets. Dense captions in fast-paced scenes were described as “overwhelming,” whereas off-screen sounds were seen as more deserving of richer detail.

Several trade-offs emerge consistently from the study. One is the trade-off between information richness and cognitive load: more descriptive captions can improve comprehension but may exceed available reading time or duplicate visible action. Another is the tension between interpretive and descriptive NSI: some viewers welcomed emotionally resonant captions, while others objected to captions that “think for me” (Huang et al., 27 Aug 2025). A third is the context dependence of preferences: documentary, cinematic, educational, and music-focused material elicited different optimal settings. The abstract summarizes these findings succinctly, stating that CapTune supported creators’ creative control while enhancing viewers’ emotional engagement, and that the evaluation revealed trade-offs between information richness and cognitive load, tensions between interpretive and descriptive representations of sound, and the context-dependent nature of caption preferences (Huang et al., 27 Aug 2025).

These findings caution against a simplistic reading of personalization as uniformly beneficial. The system’s own evidence suggests that customization is valuable precisely because preferences are unstable across viewers, scenes, and tasks.

6. Relation to Adjacent Research and Terminological Distinctions

Within the supplied literature, the name “CapTune” should be distinguished from two adjacent lines of work that address different modalities and tasks. "Masked Acoustic Unit for Mispronunciation Detection and Correction" presents a Computer-Assisted Pronunciation Training framework that uses acoustic units as an intermediary representation for mispronunciation detection and correction (Zhang et al., 2021). The supplied details explicitly frame this as a “CapTune-style CAPT product,” but the paper itself is not a system named CapTune. Its goal is speech-based self-imitating feedback for pronunciation learning rather than non-speech caption personalization.

Similarly, "KaraTuner: Towards end to end natural pitch correction for singing voice in karaoke" is an end-to-end neural pitch correction architecture for karaoke singing (Zhuang et al., 2021). The supplied details describe KaraTuner as informing a “CapTune-style” product brief because it replaces rule-based deviation estimation and DSP-heavy resynthesis with a Transformer pitch predictor and a pitch-controllable neural vocoder. Here again, the task is automatic pitch tuning from raw karaoke recordings, not caption adaptation.

The overlap across these works is therefore conceptual rather than nominal. All three involve controlled transformation of media signals under constraints: pronunciation correction constrained by target phonemes, karaoke pitch correction constrained by musical notes and vocal spectra, and non-speech caption adaptation constrained by creator anchors and scene context. This suggests a broader pattern in which “CapTune-style” design denotes bounded personalization or correction rather than unconstrained generation. However, only the 2025 work names the system CapTune and applies it to accessibility-oriented NSI customization (Huang et al., 27 Aug 2025).

7. Limitations, Misconceptions, and Future Directions

CapTune’s current parameterization is intentionally simple. The four exposed dimensions are described as a simplification of nuanced and fluid preferences, and the paper notes that linear interpolation may not capture cross-scene dynamics or changing viewer states such as fatigue or social setting (Huang et al., 27 Aug 2025). The evaluations used short clips, so questions of narrative consistency and sustained cognitive load in long-form viewing remain open.

Another limitation concerns the generative backend. Because the system relies on GPT-4o without finetuning, occasional inconsistency or semantic drift can persist despite anchored prompts and creator bounds (Huang et al., 27 Aug 2025). The paper identifies future work in finetuning, hybrid pipelines, and automatic post-hoc validation, including possible embedding-based similarity checks. It also highlights the need for larger-scale quantitative studies that measure not only preference, but also accuracy, consistency, timing, and comprehension outcomes across genres.

Two misconceptions are directly rebutted by the system design and findings. The first is that accessibility personalization should maximize descriptiveness. The studies instead show that more detail can be helpful or overwhelming depending on pace, visibility, and viewer preference. The second is that expressive NSI is universally desirable. The evaluation reports that some viewers value evocative captioning while others reject interpretive overreach, particularly when captions appear to attribute emotions or intentions not clearly grounded in the scene (Huang et al., 27 Aug 2025).

The paper’s practical recommendations follow from these constraints. Conservative bounds, scene-aware anchoring, locking of sensitive items, and support for presets or profiles are all presented as mechanisms for reconciling personalization with consistency and cognitive economy. Taken together, these design choices position CapTune not as a replacement for caption authorship, but as an infrastructure for controlled co-authorship between creators and viewers.