AnnoSense Emotion Annotation Framework

• AnnoSense is a human-centered framework for collecting in-situ physiological and mobile-sensing emotion data that emphasizes contextual richness.
• It outlines a comprehensive three-phase methodology—pre, during, and post-data collection—that integrates adaptive annotation with robust ethical protocols.
• The framework’s multi-stakeholder approach enhances data quality by grounding emotional labels through participant training, consent procedures, and multi-perspective analysis.

Searching arXiv for the AnnoSense paper and closely related annotation/anomaly papers to ground citations. arXiv search: "AnnoSense framework physiological emotion data collection everyday settings AI" AnnoSense is a human-centred framework for collecting physiological and mobile-sensing–based emotion data in everyday life for emotion AI. It is not a dataset or an algorithm; rather, it specifies an end-to-end methodology for pre-data collection, during data collection, and post-data collection, with the explicit aim of improving the quality, contextual richness, and ethical robustness of emotion annotations obtained through wearables and smartphones in real-world settings (Singh et al., 17 Jul 2025).

1. Definition, scope, and problem setting

AnnoSense was proposed in response to a central methodological difficulty in emotion AI: algorithms require high-quality data and accurate annotations, yet everyday emotion data collection is intrinsically difficult because self-reports are subjective, physiological signals are noisy and person-specific, and real-world annotation imposes burden, fatigue, and privacy costs (Singh et al., 17 Jul 2025). The framework is therefore explicitly data-centric and stakeholder-driven.

Its scope is narrower and more precise than generic annotation infrastructure. AnnoSense targets physiological emotion data collection in everyday settings for AI, especially wearable- and smartphone-based studies, whereas systems such as audino are collaborative annotation tools for audio and speech with temporal segmentation, role management, and JSON export (Grover et al., 2020). This distinction is important: AnnoSense is a procedural framework for study design, participant management, annotation strategy, data handling, and post hoc grounding, not a software package for a single modality.

The framework is motivated by limitations in both laboratory and in-the-wild affective datasets. Laboratory datasets such as WESAD, ASCERTAIN, CASE, AMIGOS, and RECOLA are described as relying on artificial stimuli and rigid scales with little room for context. Semi-naturalistic or real-life datasets such as StudentLife, GLOBEM, LAUREATE, K-EmoPhone, TILES, SWEET, and DAPPER are described as using ESM/EMA and end-of-day diaries but facing low response rates, annotation fatigue, reactivity, and limited contextual detail about triggers, social situation, and personal meaning (Singh et al., 17 Jul 2025).

At the conceptual level, AnnoSense treats emotion annotation as a socio-technical process rather than a simple label-acquisition problem. The framework assumes that emotional literacy, stigma, coping style, daily routine, device comfort, and trust all affect the validity of annotations. A plausible implication is that AnnoSense shifts the center of gravity of emotion AI from model design toward the construction of grounded and usable labels.

2. Empirical basis and development process

AnnoSense was derived from three empirical studies involving 119 stakeholders in total: 75 survey responses, 32 interviews with members of the public, and 3 focus group discussions with 12 mental health professionals (Singh et al., 17 Jul 2025). These stakeholder perspectives were then used to formulate the framework and were followed by an evaluation by 25 emotion AI experts for clarity, usefulness, and adaptability.

The survey examined emotional awareness, emotional vocabulary and granularity, conceptual understanding of emotion and emotional intensity, existing emotion-management practices, and attitudes toward annotation. Reported findings included substantial diversity in emotional literacy and annotation preferences, a strong desire for context-sensitive and flexible methods, and reservations about daily tracking and negative-emotion logging (Singh et al., 17 Jul 2025).

The interviews were semi-structured and organized with a 5W1H framing. They surfaced several recurrent themes: many participants described avoidance and distraction as common ways of dealing with emotions, especially negative ones; stigma around emotional disclosure remained strong; participants wanted agency over prompts; and many preferred some tangible return, such as feedback or insights, for the effort of tracking (Singh et al., 17 Jul 2025).

The focus groups contributed a clinical and psychosocial perspective. Mental health professionals emphasized that emotions are contextual and constructed, that no single physiological pattern maps unambiguously to a discrete emotion, and that useful assessment depends on history taking, psycho-social context, emotional range, congruency, intensity, reactivity, vocabulary, and sometimes multiple perspectives, including patient, family, and clinician input (Singh et al., 17 Jul 2025). They also argued that AI should be treated as a parallel source of information rather than a replacement for clinical judgment.

The framework was produced by thematic analysis of survey, interview, and focus group materials, followed by synthesis into three stages of data collection. Those stages were aligned with prior work in data-centric AI, EMA/ESM practice, critiques of self-report, and emotion AI ethics, yielding 15 actionable guidelines grouped into pre-data collection, during data collection, and post-data collection (Singh et al., 17 Jul 2025).

3. Framework structure and the fifteen guidelines

AnnoSense is organized into three phases: pre-data collection, during data collection, and post-data collection. The phases are sometimes summarized as “Two-way communication,” “Understanding the needs of the data source,” and “Learning from dynamic data” (Singh et al., 17 Jul 2025).

Phase	Guidelines	Focus
Pre-data collection	G1	Selecting participants
Pre-data collection	G2	Obtaining informed consent
Pre-data collection	G3	Initial calibration
Pre-data collection	G4	Participant training
Pre-data collection	G5	Psycho-social profiling
Pre-data collection	G6	Demographic and medical data
During data collection	G7	Focus on participant’s agency
During data collection	G8	Participant-aware sampling
During data collection	G9	Adaptable annotation methods
During data collection	G10	Multi-perspective assessments
During data collection	G11	Engagement, learning, and support
Post-data collection	G12	Secure data handling
Post-data collection	G13	Data quality validation and pre-processing
Post-data collection	G14	Holistic analysis and grounding of emotion data
Post-data collection	G15	Sharing findings, limitations, and usability

In the pre-data collection phase, the framework begins with participant selection and screening. G1 specifies inclusion and exclusion criteria aligned with study aims, emphasizes demographic diversity, and recommends screening for alexithymia using validated scales such as TAS-20 or the Perth Alexithymia Questionnaire, as well as for neurological and medical conditions that may affect physiological signals or emotional processing. G2 addresses informed consent, including risks, benefits, anonymization, storage, legal compliance, and data-sharing strategy. G3 and G4 cover baseline calibration, device familiarization, correct wearable use, practice labeling, differentiation of similar emotions, and explicit instruction in documenting context. G5 and G6 extend preparation beyond conventional demographics by incorporating psycho-social profiling, emotional range, congruency, intensity, reactivity, coping strategies, socio-family context, and other relevant medical or personality variables (Singh et al., 17 Jul 2025).

The during-data-collection phase is centered on reducing burden while improving annotation quality. G7 foregrounds participant agency through lightweight and unobtrusive wearables and adjustable annotation frequency. G8 proposes participant-aware sampling based not only on conventional context-aware EMA triggers such as location or movement, but also on daily schedule, physiological changes, and individual emotional profile. G9 advocates adaptable annotation methods that combine structured formats such as SAM, Likert scales, PANAS items, emotion lists, and valence-arousal sliders with unstructured forms such as free text, audio notes, images, and sketches. G10 introduces multi-perspective assessments by permitting input from close others, caregivers, or clinicians and by combining self-reports with other sensed streams. G11 focuses on maintaining engagement, building emotional literacy, and providing support through periodic check-ins, visual feedback, psychoeducational resources, and optional links to professional help (Singh et al., 17 Jul 2025).

The post-data-collection phase treats emotion data as sensitive, dynamic, and interpretively layered. G12 covers secure storage, de-identification, and participant review or deletion rights within a defined time window. G13 addresses missing data, artifacts, inconsistencies, normalization, and cross-validation across modalities. G14 is the most methodologically distinctive component: it recommends combining qualitative and quantitative evidence to produce multi-dimensional emotion labels grounded in subjective experience, context, triggers, physiological activation, and social impact, while also weighing source reliability and integrating psycho-social profiles. G15 requires transparent reporting of protocol details, challenges, limitations, intended AI use cases, and the actual suitability of the collected data for those use cases (Singh et al., 17 Jul 2025).

4. Annotation model, sensing modalities, and grounding logic

AnnoSense does not prescribe a single annotation schema. Instead, it recommends adaptive, layered annotation in which the annotation method varies with emotional intensity, time availability, and participant preference (Singh et al., 17 Jul 2025). This is a methodological departure from one-size-fits-all prompting.

Structured labels may include dimensional ratings such as valence and arousal, categorical emotion labels, intensity ratings, and duration estimates. The framework explicitly allows multiple simultaneous emotions and encourages richer vocabularies, for example through emotion wheels and curated lists. Unstructured inputs may include free text, audio, images, and sketches. For richer subjective reflection, AnnoSense suggests scaffolded formats such as the ABC model and CBT-style thought records, and it also discusses LLM-based guided prompts tuned by emotional traits identified during profiling (Singh et al., 17 Jul 2025).

The sensing stack assumed by the framework is typical of ubiquitous and wearable computing. Physiological streams may include PPG/HR, HRV, EDA, skin temperature, respiration, accelerometer, and gyroscope. Mobile sensing may include GPS, app usage, screen on/off, and communication logs, subject to governance and participant choice (Singh et al., 17 Jul 2025). Smartphones are treated as the primary surface for quick reports, rich entries, and feedback, while wearables support continuous sensing and occasional lightweight prompts.

A central methodological claim of AnnoSense is that ground truth cannot be reduced to a single subjective label detached from context. G14 therefore recommends grounding emotion data holistically by combining qualitative and quantitative evidence and by weighting sources according to relevance and reliability. Self-report remains primary for inner experience, but expert judgments may carry more weight for clinical dysregulation and peers may be informative for social-conflict dynamics (Singh et al., 17 Jul 2025). This suggests a shift from “emotion label as a scalar target” toward “emotion label as a constructed, multi-source object.”

Timing is similarly treated as adaptive rather than fixed. The framework allows time-based prompts, event-contingent or sensor-triggered prompts, and participant-initiated reports. It also allows mixed strategies that combine in-the-moment entries with daily or weekly reflection, informed by passively collected context. The stated rationale is to balance recall bias against burden and to avoid prompt schedules that are technically convenient but behaviorally counterproductive (Singh et al., 17 Jul 2025).

5. Stakeholders, ethics, and governance

AnnoSense explicitly identifies three stakeholder groups: everyday participants or data subjects, mental health professionals, and emotion AI researchers and developers (Singh et al., 17 Jul 2025). The framework is built around balancing their needs rather than privileging any single constituency.

Participants are treated as active co-annotators and co-interpreters rather than passive data sources. This is reflected in recommendations on participant training, control over prompts, choices about which streams are collected, control over multi-party input, and access to feedback or learning resources. A plausible implication is that AnnoSense treats adherence and honesty not as fixed personal traits but as outcomes of design.

Mental health professionals contribute construct validity and practical caution. Their role appears most strongly in psycho-social profiling, label grounding, and the insistence that physiological emotion data should not be read as a direct proxy for diagnostic truth. The framework adopts the clinical view that AI outputs are supporting signals and that actionability depends on congruence across sources rather than on an imagined perfectly objective label (Singh et al., 17 Jul 2025).

Emotion AI researchers and developers are assigned responsibility for protocol design, instrumentation, preprocessing, and transparent reporting. This includes careful documentation of device limitations, synchronization between sensor streams and annotation timestamps, normalization decisions, and downstream use cases (Singh et al., 17 Jul 2025).

Ethics is not isolated into a single section of the framework; it is distributed across several guidelines. G2 addresses informed consent, including legal compliance such as GDPR and HIPAA. G12 addresses encryption, de-identification, deletion rights, and the limits of deletion after aggregation or anonymization. G7, G8, and G10 address autonomy over prompts, sensing, and contributor access. G11 addresses potential harms of annotation itself, including rumination, distress, and reinforcement of negative affect, and recommends balancing reflection on difficult states with support for positive reflection and coping (Singh et al., 17 Jul 2025).

The framework also responds directly to mistrust and stigma. Participants in the empirical studies reportedly feared misuse of emotion data, including misdiagnosis or inappropriate use in work and social-media settings. AnnoSense’s response is to emphasize transparency, participant control, and the avoidance of simplistic diagnostic interpretations (Singh et al., 17 Jul 2025).

6. Evaluation, relation to prior work, and open questions

AnnoSense was evaluated by 25 experts in emotion AI, affective computing, HCI, and ubiquitous computing, who rated each guideline for clarity, usefulness, and adaptability on 5-point Likert scales (Singh et al., 17 Jul 2025). The reported result was broadly positive: most guidelines were judged “good” or “excellent” in clarity and usefulness; adaptability was somewhat more mixed; and no guideline received “poor” ratings in any dimension. Expert feedback requested more examples, clearer definitions, and acknowledgment that some steps, especially psycho-social profiling, may not be feasible in all contexts. The framework was subsequently refined to clarify terms such as alexithymia and to improve examples and wording (Singh et al., 17 Jul 2025).

Relative to prior work, AnnoSense advances beyond emotion representation schemes and annotation tools by binding technical, psychological, and ethical considerations into a single study-design framework. In contrast to lab-centered affective datasets, it prioritizes ecological validity and context. In contrast to conventional EMA/ESM deployments, it recommends participant-aware prompting and flexible annotation depth. In contrast to single-source labeling paradigms, it explicitly supports multi-perspective assessment and post hoc holistic grounding (Singh et al., 17 Jul 2025).

Its relationship to annotation tooling is complementary rather than competitive. Tools such as audino provide centralized user management, temporal segmentation, configurable labels, and API-based data upload for audio and speech annotation (Grover et al., 2020). AnnoSense instead specifies how participant screening, consent, adaptive prompting, emotion literacy training, psycho-social profiling, secure storage, and multi-source grounding should be organized for everyday emotion AI (Singh et al., 17 Jul 2025). This suggests that AnnoSense could, in principle, be operationalized through software systems, but it is not itself such a system.

Several limitations remain explicit. The framework acknowledges that psycho-social profiling and clinician involvement may be difficult to scale, that the empirical work was conducted within one country and therefore requires cross-cultural validation, and that the kinds of dynamic, multi-layered labels it recommends will require new modeling approaches beyond simple categorical or continuous targets (Singh et al., 17 Jul 2025). It also raises unresolved questions about fairness, institutional integration, and the risks of surveillance or over-interpretation.

In the literature of emotion AI, the significance of AnnoSense lies less in proposing a new sensing modality or learning architecture than in codifying a rigorous methodology for constructing better supervision. Its central claim is that everyday emotion data suitable for AI cannot be produced by combining passive sensing with sparse self-report alone; it must be built through participant preparation, adaptive and context-rich annotation, careful governance, and post hoc grounding across multiple sources (Singh et al., 17 Jul 2025).