DescribePro: Collaborative Audio Description System
- DescribePro is a collaborative audio description system that combines AI-generated baselines with iterative human refinement.
- The platform integrates modules for timing detection, multimodal prompting, manual editing, forking variations, and tagging to enhance workflow efficiency.
- Evaluations indicate that DescribePro effectively supports both professional and novice users by balancing rapid generation with precise stylistic control.
Searching arXiv for the DescribePro paper and nearby work to ground the article. DescribePro is a collaborative audio description authoring system for human–AI interaction in which describers iteratively refine AI-generated descriptions through multimodal LLM prompting and manual editing, while also supporting community collaboration through forking, variation management, and tagging (Cheema et al., 1 Aug 2025). It is presented as a response to the trade-off between the precision of human-crafted audio descriptions and the efficiency of AI-generated ones. The system is evaluated with 18 describers, including 9 professionals and 9 novices, and is designed so that final output comprises one or more tagged variations, each with time-aligned audio description scripts ready for recording or synthesis (Cheema et al., 1 Aug 2025).
1. Scope and authoring model
Audio description makes video content accessible to millions of blind and low vision users, but high-quality authoring requires both temporal alignment and stylistic control (Cheema et al., 1 Aug 2025). DescribePro addresses this by combining automatic baseline generation with iterative human revision. Its central authoring model is collaborative rather than fully automatic: AI generates an initial set of time-aligned descriptions, and describers then refine them through prompt-based revision, manual editing, timing adjustment, and branching into alternative narrative styles.
The system is explicitly organized around the coexistence of professional and novice workflows. Professionals can preserve stylistic choices through precise prompts and manual overrides, while novices can begin from AI baselines and existing variations rather than from a blank slate. This suggests that DescribePro is not framed as a replacement for describers, but as an environment for controlled augmentation of describer labor, with branching structures for reuse, comparison, and quality-control passes (Cheema et al., 1 Aug 2025).
2. System architecture and end-to-end workflow
DescribePro is a web-based platform built around five core modules (Cheema et al., 1 Aug 2025).
| Module | Function | Technical components |
|---|---|---|
| AD Timing Module | Detects optimal intervals for inserting audio descriptions | Pydub, Silero VAD, PySceneDetect |
| Description Generation Module | Produces an initial set of time-aligned descriptions | frame sampling, 42 general AD guidelines, GPT-4o |
| AI Prompting & Editing Interface | Supports iterative revision and manual editing | timeline overlay, prompt box, diff view |
| Forking Interface | Creates and manages alternative variations | metadata, previewing, parent variation linkage |
| AD Tags Module | Assigns style/content tags for filtering and cues | GPT-4o, predefined and custom tags |
The AD Timing Module analyzes the uploaded video’s audio, distinguishing silence versus non-speech via Pydub and Silero VAD, and analyzes visuals through scene changes via PySceneDetect to detect optimal intervals for inserting audio descriptions. The Description Generation Module samples video frames, specifically one every two seconds between each audio-description timestamp, merges user-provided custom instructions with a curated set of 42 general AD guidelines, and invokes GPT-4o to produce an initial set of time-aligned descriptions (Cheema et al., 1 Aug 2025).
The workflow is specified as a seven-step sequence. A user uploads a video; the AD Timing Module identifies candidate insertion intervals; frames are sampled and batched for GPT-4o, which generates baseline descriptions using the combined prompt of custom instructions and the 42 AD guidelines; the AI-generated descriptions and initial tags are saved to the database; describers review and refine them via the AI Prompting & Editing Interface or manual edits; the Forking Interface allows creation of alternative variations with version lineage and tagging; and the final output consists of one or more tagged variations ready for recording or synthesis (Cheema et al., 1 Aug 2025).
3. Multimodal prompting, revision, and editing controls
DescribePro uses multimodal LLM prompting with three input modalities: video frames, optional transcripts when users prompt for “Text on screen” transcription, and the combination of user custom instructions with the system’s 42 general AD guidelines (Cheema et al., 1 Aug 2025). The baseline generation prompt instructs GPT-4o to produce descriptive, objective, clear audio descriptions given a sequence of images, emphasizes user-provided guidelines over general ones, and asks for explicit mention of any overlooked guidelines for transparency. The revision prompt is formulated as: “You are an advanced AI designed to enhance and refine audio descriptions… Input Prompt: {user’s instruction} Description: {original description} You will also be provided with relevant video frames…”, and it returns only the revised description. A separate tagging prompt asks GPT-4o to select up to four keywords plus up to two custom tags from predefined tag categories.
The revision interface supports iterative batch edits. Describers can select multiple timestamps and apply one natural-language prompt across all selected segments, with examples including “Remove all emotional interpretations” and “Shorten descriptions by 30%.” For each batch, previously accepted descriptions are included in the prompt to reduce repetition. Revised descriptions appear side-by-side with originals, with insertions and deletions highlighted in-line, enabling quick accept or reject decisions (Cheema et al., 1 Aug 2025).
Editing is not restricted to prompt-based interaction. The interface includes a prompt box for global or targeted edits, a manual text editor for copy-paste, in-place corrections, deletions, and reordering of individual descriptions, timestamp controls to split, merge, or shift description timing if auto-detected intervals are slightly off, and accept or reject controls at the description level or in bulk. Each prompt submission and AI response is saved, along with user acceptance and rejection logs. Users can revisit prior prompts, compare multiple AI versions, and re-issue refined prompts. The system also records prompt types such as “Add detail,” “Remove redundancies,” and “Switch to active voice” for analysis and potential future recommendation (Cheema et al., 1 Aug 2025).
A key design element is preservation of professional stylistic choices. Prompt templates prioritize user-provided stylistic guidelines over generic rules. Professionals can fine-tune tone, conciseness, and voice through precise prompts such as “Make passive sentences active,” “Remove adverbs,” and “Use formal register,” while manual overrides ensure that any AI-generated content can be edited back into a describer’s unique style without loss of control (Cheema et al., 1 Aug 2025).
4. Forking, tags, and collaborative variation management
DescribePro extends beyond single-author editing through a forking model in which any variation, whether AI-authored or human-authored, can be forked (Cheema et al., 1 Aug 2025). When a variation is forked, the system clones timestamps, text, and tags, increments fork count, and links to the parent variation for lineage. Contributors are attributed, and users can inspect how many times a variation has been forked. The interface displays a navigation bar of all variations, their tags, fork count, and source lineage, and users can preview variations side-by-side and fork from any point to explore alternative tones or detail levels.
The tags system combines predefined categories and custom annotations. DescribePro automatically assigns up to four predefined style or content tags, with examples including “Concise,” “Character focus,” “With Interpretations,” and “High detail,” plus up to two custom tags via GPT-4o. Tags can be edited by users and serve as filters or narrative cues. Predefined categories include Length, Focus, Interpretations, Detail Level, and Action, Character, or Environment emphasis. The paper states that tags can guide both AI generation through custom prompts and end-user consumption, for example when blind and low vision users select a “concise” versus “detailed” track (Cheema et al., 1 Aug 2025).
The collaborative design has two distinct functions. First, it supports custom narrative styles, including alternative tones such as optimistic versus melancholic, or alternative emphases such as environment-only or character-only descriptions. Second, it supports version control and pedagogy: professional teams can track quality-control passes via successive forks, and novices can learn by browsing professional and peer variations. This suggests that collaboration is encoded not merely as multi-user access, but as an explicit versioned representation of stylistic alternatives (Cheema et al., 1 Aug 2025).
5. Evaluation protocol and collected measures
The system was evaluated with 18 describers, specifically 9 professional and 9 novice participants, recruited via online groups and snowball sampling; professionals averaged 12+ years of paid audio-description experience (Cheema et al., 1 Aug 2025). The study procedure included orientation and demographic survey, three audio-description tasks in counterbalanced order, a post-task survey with the System Usability Scale and feature usefulness ratings, and a semi-structured interview for qualitative feedback.
The three tasks were defined as follows: Task 1 required refinement of AI-generated descriptions; Task 2 required review and revision of one of three researcher-created human variations; and Task 3 was open-ended, with participants choosing one of four videos and either forking an existing variation or starting fresh with AI (Cheema et al., 1 Aug 2025). Two 3–4 minute videos, a short film and a cooking tutorial, were used for Tasks 1 and 2, while four varied clips—an origami tutorial, a dog show, a scenic tour, and another short film—were offered in Task 3.
The evaluation collected interaction logs and multiple textual similarity measures. Logged data included time on task, number and type of prompts, accept or reject rates, timing adjustments, edit counts, Levenshtein distances, similarity metrics, System Usability Scale scores, usefulness ratings, and interview transcripts (Cheema et al., 1 Aug 2025). Similarity analysis included lexical similarity using SeqMatcher, semantic similarity using BGE, and stylistic similarity using LUAR-MUD. The breadth of these measures indicates that the evaluation was not limited to usability alone, but also examined how much participants transformed AI and human baselines.
6. Quantitative and qualitative findings
The quantitative results report mean SUS = 72.6 with SD = 13.9, above the 68 benchmark for web interfaces (Cheema et al., 1 Aug 2025). “Forking” was rated the most useful feature, while “Tags” was rated the lowest. Novices valued AI baselines and forking more, whereas professionals valued the prompting interface. Professionals issued 82 AI prompts, of which 72.8% were accepted; novices issued 65, of which 60.1% were accepted. Professionals favored language refinement and removal prompts, while novices favored addition prompts.
The paper also reports a mixed ANOVA showing a significant task-type effect on word-count change, given as , , , and states that participants cut human-generated descriptions more than AI baselines (Cheema et al., 1 Aug 2025). Average Levenshtein distance was 136.7 edits for human variations versus 51.4 edits for AI variations. Timing adjustments affected 11.4% of timestamps, with an average shift of approximately 0.8 s. Task duration was similar across AI-variation and human-variation edits, specifically min with SD for AI-variation edits and min with SD for human-variation edits. Novices’ edits were more similar to originals than professionals’ across lexical, semantic, and stylistic measures: SeqMatcher .650 versus .580, BGE .882 versus .837, and LUAR-MUD .866 versus .826.
The qualitative findings are organized into four themes (Cheema et al., 1 Aug 2025). The first theme, easing the tediousness of audio-description authorship, reports that AI baselines gave a head start and reduced the cognitive load of a blank slate; professionals appreciated offloading repetitive tasks such as on-screen text transcription; novices gained confidence and noticed visual details they might have missed; and concerns over AI hallucinations and subjectivity emerged, although most participants saw AI as a collaborator rather than a replacement. The second theme, editing with AI—expectations and limitations, reports that bulk edits via prompts were efficient for general changes in tone, length, and detail, while precise single-segment edits were often easier manually; prompt formulation had a learning curve, and participants suggested word-level accept or reject controls and a “refine this revision” option.
The third qualitative theme distinguishes professional and novice workflows. Professionals focused on language refinement, conciseness, and narrative flow, often leveraging AI for grammar and tone edits. Novices focused on adding detail and were more likely to query the AI for context, including identifying text and objects. The fourth theme concerns collaboration and customization via variations: forking enabled multiple narrative tracks, professionals used forks for version control across quality-control passes, novices used them as learning examples, and tags were under-utilized but recognized as valuable filters for describers and end users (Cheema et al., 1 Aug 2025).
7. Limitations, design trade-offs, and future directions
The paper characterizes the central trade-off directly: AI accelerates baseline generation and bulk edits, but human precision remains essential for nuance, emotional tone, and domain-specific research (Cheema et al., 1 Aug 2025). It further states that a hybrid AI-human workflow reduces tedium and supports scalability while preserving stylistic agency; that granular prompting and manual override are both necessary, since professionals demand fine-grained control and novices benefit from guided assistance; and that collaborative branches and tags foster personalization and peer learning.
The reported limitations are also explicit. The study involved small, one-hour lab sessions with 18 describers and lacked longitudinal or in-the-wild deployment. There was no direct evaluation of final audio descriptions with blind and low vision end users. The paper also notes that cognitive load may rise with many variations, implying a need for filtering and recommendation systems (Cheema et al., 1 Aug 2025). This suggests that the versioned collaboration model, while useful for customization and training, may produce navigational overhead as the number of branches grows.
Future directions listed in the paper include integrating blind and low vision user feedback loops for real-time quality assessment and preference learning, developing genre- and region-specific audio-description guideline adapters, offering tiered modes for novice, professional, and blind and low vision contributors with tailored AI assistance and feedback, and exploring hybrid AI models in which smaller fine-tuned engines handle edits and large multimodal models handle scene understanding in order to reduce environmental impact (Cheema et al., 1 Aug 2025). A plausible implication is that subsequent work may treat DescribePro not only as an authoring interface, but also as a framework for adaptive, audience-conditioned audio-description production.