Participant-Narrated Annotations

• Participant-narrated annotations are a method where involved individuals provide natural language labels that serve as rich, context-aware ground-truths.
• They utilize multi-stage frameworks and structured schemas to capture goal-oriented narratives, counterfactuals, and subjective rationales.
• Applications range from narrative event analysis and web renarration to robotics, enhancing multimodal understanding and user-driven system design.

Participant-Narrated Annotations refer to annotation methods in which people directly involved with or observing an event, artifact, or process—that is, participants rather than external annotators or system designers—generate natural language (or multimodal) explanations, descriptions, or alternative renderings. These annotations can serve as ground-truth labels, subjective rationales, counterfactuals, or personalized translations, capturing perspectives, intentions, or experiences that traditional annotation schemes often miss. The following sections detail foundational definitions, representative methodologies, application domains, system architectures, and research outcomes in participant-narrated annotation, with an emphasis on recent developments in narrative understanding, multimodal grounding, and human-computer interaction.

1. Core Definitions and Theoretical Foundations

Participant-narrated annotation is characterized by the production of annotation content—labels, explanations, stories, or transformations—by individuals who are stakeholders or agents in the relevant scenario. Unlike conventional expert-annotated or crowd-labeled datasets, participant-narrated annotations foreground the goals, intentions, and experiences of participants as the epistemic source.

In narrative event analysis, this can mean taking a “participant achievement lens”, whereby for each volitional agent in a story, annotators specify (i) the super-ordinate goal pursued, (ii) a likely next action, and (iii) their estimation of goal success (Vallurupalli et al., 2024). In web content accessibility, renarrations represent user-authored contextual adaptations—simplifications, translations, elaborations, or audio/video retellings—tagged with metadata about intended audiences and transformation types (Dinesh et al., 2018). In multimodal data and video, participant-narrated annotations include spoken or written descriptions produced synchronously or asynchronously by actors within the scenario, often temporally aligned to sensor traces, video, or physiological signals.

This paradigm distinguishes itself from purely observer-based labeling by capturing subjectivity, context, and lived experience, thus supporting richer representation of intention, satisfaction, counterfactuality, and alternative narrative trajectories.

2. Annotation Schemes and Workflows

Implementation of participant-narrated annotation methods varies by domain and task:

• Narrative Event Understanding (SAGA): The SAGA framework structures annotations in a multi-stage Amazon Mechanical Turk pipeline: annotators generate free-form goal statements, rate goal success and satisfaction (Likert scale), predict future actions, provide explanations, and, when applicable, propose revised plans for goal attainment. Alternate story continuations are minimally edited to support counterfactual reasoning, with successive annotation rounds for coherence, inferability, and plan adaptation (Vallurupalli et al., 2024).
• Web Renarration: Renarration uses the W3C Web Annotation Data Model, extending the standard annotation triple (target, body, motivation) with an explicit “renarration” motivation and properties detailing audience profile and transformation type. Authors of renarrations inject alternative, context-sensitive content (text, audio, video) that replaces or supplements original fragments for users matching specified profiles (Dinesh et al., 2018). The formal model is:

$\text{REN} = (T, B, m, p)$

where $T$ is the target, $B$ is the alternative body, $m$ the motivation (renarration), and $p$ the property set (audience, transformation type, etc.).

• Emotion and Sensor Data: In emotion annotation from VR/physiology, participants are prompted for both structured (scales) and narrative self-reports, often in semi-structured interviews immediately following stimulus exposure. Annotation alignment is assessed quantitatively (Wilcoxon test; Cohen's kappa) and qualitatively via thematic analysis of narratives (Singh et al., 25 Mar 2025). In wearable activity recognition, in situ and retrospective (with or without time-series visualization) self-annotation protocols are deployed, balancing temporal precision, annotation completeness, and cognitive load (Hoelzemann et al., 2023).
• Audio and Multimodal Texts: In citizen science phonetic annotation tasks, end-users—often non-experts—generate corrections, completions, or transcriptions at graded levels of scaffolding, with evaluation metrics such as phonologically weighted Levenshtein distance quantifying annotation accuracy (Foradi et al., 2019).

3. Formal Representation and Data Structures

Participant-narrated annotations commonly adopt data models supporting rich metadata and hierarchical, composable representations:

• Web Annotation Model: Annotations are encapsulated using RDF/JSON-LD triples linking source content and annotation body, with motivation and transformation type. A typical instantiation for renarration appears as: $T$0 (Dinesh et al., 2018).
• Narrative Annotation Categories (SAGA): Annotation schemas include fields for volitional participant identification, goal description (free text), goal success (5-point Likert), next action prediction, explanations (free text), directionality, satisfaction, plan revision likelihood, future plan (original/revised), and alternate story coherence/inferability (Vallurupalli et al., 2024).
• Multimodal/Sequential Data: Multistream alignment – synchronizing narrative text, action/event logs, and video/audio – provides temporally grounded multimodal demonstration traces for program synthesis or reward learning (Kim et al., 29 Sep 2025, Tung et al., 2018).

4. Application Domains and Experimental Findings

Participant-narrated annotation frameworks have enabled advances across diverse research areas:

• Complex Event and Narrative Reasoning: The SAGA dataset demonstrates that participant-centric goal and action annotations expose subtleties in narrative understanding, with high inter-annotator agreement ($\kappa = 0.80$) on nuanced, multi-faceted tasks. Benchmarks show that fine-tuned small LMs can outperform much larger frozen LMs in intent and plan inference (Vallurupalli et al., 2024).
• Web Accessibility and Social Inclusion: Renarration systems extend content reach to underrepresented audiences by incorporating alternative renderings, improving accessibility for linguistic, literacy, or cultural minority groups, and decentralizing narrative authority (Dinesh et al., 2018).
• Multimodal Perception and Robotics: Narrated video and audio annotations enable self-supervised and weakly supervised learning of perceptual grounding, action policies, and semantic representations. Examples include:
  • Reward detectors for robotics: Learning binary and shaped rewards for pick-and-place via direct narrative grounding (Tung et al., 2018).
  • Contextualized object embeddings: Matching visual features to contextualized language representations learned from narrated video streams for robust few-/zero-shot learning (Bertasius et al., 2020).
  • Program synthesis from narrated demonstrations: Automatic construction of behavioral finite-state machines from multimodally grounded participant narratives, enabling end-user correction and iterative repair (Kim et al., 29 Sep 2025).
• Data Visualization and Collective Storytelling: Embedding audience-authored annotations into visualizations allows shared personal narratives to supersede designer-imposed narratives, supporting sensemaking and collective memory in data exploration (Kauer et al., 24 Mar 2025).

5. Bias, Agreement, and Evaluation

Empirical studies reveal both strengths and limitations of participant-narrated annotation approaches:

• Reliability: When designed with clear guidelines and example-driven training, participant-narrated annotations support high inter-annotator agreement, even with complex or free-form tasks (e.g., SAGA: $\kappa=0.80$; program synthesis correction: completeness improvement $\Delta D=+16.7\%$, $p=0.03$) (Vallurupalli et al., 2024, Kim et al., 29 Sep 2025).
• Systematic Bias: Annotation modality impacts both empirical label quality (#missing, precision) and downstream model performance (F1). In HAR tasks, in situ methods offer greater temporal accuracy but higher omission rates; retrospective visualizations close the gap (Hoelzemann et al., 2023). In emotion labeling, intrinsic (self-image, experience) and extrinsic (design, context) factors drive annotation variance (Singh et al., 25 Mar 2025).
• Consistency: Statistical measures, such as Cohen’s $\kappa$, are used to assess agreement across labeling methods, revealing that hybrid protocols can approach the precision and recall of high-friction in situ protocols (Hoelzemann et al., 2023).

6. Challenges and Future Directions

Current research identifies several persistent challenges:

• Alignment and Ground Truth: Narrated annotations are often only loosely synchronized with the underlying event or data (e.g., noisy ASR, rough timestamps, loosely aligned actions), necessitating statistical or optimal transport-based alignment (Flanagan et al., 2023, Chen et al., 2023).
• User Burden and Cognitive Load: Enhanced annotation expressivity (free-form, multimodal, narrative) imposes greater user effort, mitigated through progressive scaffolding, visualization tools, or deferred annotation with memory aids (Hoelzemann et al., 2023, Foradi et al., 2019).
• Expressivity and Modality: Many annotation schemas are bounded by the expressivity of the API/library or data model (e.g., missing action primitives, limited constraint specification), highlighting the need for adaptive schema and API evolution based on emergent usage patterns (Kim et al., 29 Sep 2025).
• Bias and Generalizability: Subjectivity and prior experience influence annotation content. Integrating participant context, pre-study profiling, and hybrid supervision may attenuate uncontrolled variance (Singh et al., 25 Mar 2025).
• Integration with Downstream Tasks: While large-scale narration-driven supervision has advanced retrieval, localization, and reasoning tasks, models often underperform on generalization to minimally edited or counterfactual contexts, suggesting active learning and iterative refinement as promising directions (Vallurupalli et al., 2024).

7. Broader Implications and System Design Principles

Best practices and design recommendations for robust participant-narrated annotation systems include:

• Multi-stage pipelines to capture both retrospective and prospective reasoning (goals, actions, plans, counterfactuals) (Vallurupalli et al., 2024).
• Combination of free-form rationales and quantitative discrete-label judgments to ensure data richness and consistency (Vallurupalli et al., 2024, Singh et al., 25 Mar 2025).
• Use of minimally edited alternatives or counterfactual scenarios to probe inference boundaries and model adaptation (Vallurupalli et al., 2024).
• Provision of programmatic or graphical inspection, correction, and feedback tools to support user-driven repair, especially in interactive and program-synthesis use cases (Kim et al., 29 Sep 2025).
• Open, interoperable, and decentralized data models for annotation storage, to ensure inclusiveness, transparency, and control by both annotators and end-users (Dinesh et al., 2018).

By foregrounding participants as co-constructors of annotation meaning, this paradigm enriches datasets for narrative understanding, multimodal grounding, and interaction research, while surfacing both technical and epistemic challenges for annotation science.