I-CONECT Dataset: Multi-Modal MCI Analysis

• I-CONECT dataset is a comprehensive multi-modal collection designed to analyze mild cognitive impairment using longitudinal video, audio, and transcript data.
• It features high-frequency video chat sessions (four times per week over six months) capturing naturalistic conversations across 161 daily-life themes with rigorous clinical labeling.
• The dataset underpins benchmark studies in vision-based and linguistic MCI detection, supporting innovative deep learning models and loss functions for improved diagnostic accuracy.

The I-CONECT (Internet-Based Conversational Engagement Clinical Trial) dataset is a suite of large-scale, multi-modal resources created to enable automated analysis of mild cognitive impairment (MCI) in older adults. Originating from a randomized controlled trial (ClinicalTrials.gov NCT02871921) aimed at enhancing cognitive reserve in socially isolated adults aged 75 or older via frequent semi-structured video chat, the dataset includes longitudinal video, audio, and transcript modalities. I-CONECT is distinguished by its high session frequency (four video interviews per week over six months), naturalistic conversational context, and rigorous ground-truth cognitive labels. The dataset has served as a benchmark for deep learning research in vision-based, linguistic, and multi-modal MCI detection.

1. Composition and Participant Demographics

Cognitive and Demographic Characteristics

• Participant sets: The main cohort includes 186 older adults (≥75 years) recruited in Portland, Oregon and Detroit, Michigan: 100 diagnosed with MCI and 86 with normal cognition (NC) (Sun et al., 2023). Several subcohorts appear in domain-specific studies, e.g., 68 subjects (34 MCI, 34 NC) for linguistic analysis (Fard et al., 2024), and 69 participants for facial features research (Alsuhaibani et al., 2023).
• Gender balance: Modest male–female variation across conversation themes; for example, “Crafts Hobbies” (n = 32 videos) includes 11 males and 21 females (20 MCI, 12 NC) (Sun et al., 2023). Theme-level splits for other subsets are closely balanced.
• Age range: All subjects are ≥75; mean ages per theme typically 80–81 years (Alsuhaibani et al., 2023).
• Educational background: Means range from 15.3 to 15.8 years (per theme, SD ≈ 2.5) (Alsuhaibani et al., 2023).
• Session protocol: Each subject participated in 30-minute video chat sessions four times per week over six months, each regarding one of 161 daily-life themes (e.g., “Summertime,” “Military Service”) (Alsuhaibani et al., 2023, Fard et al., 2024).

2. Data Modalities and Collection Protocol

Video and Audio Recordings

• Recording configuration: Sessions were conducted using study-provided tablets or webcams deployed in participants’ homes, with unconstrained environmental conditions (variable lighting, backgrounds, camera angles, and occlusion) (Alsuhaibani et al., 2023).
• Session structure: Each video records a two-participant split-screen interaction, typically ~30 minutes after removal of preamble/conclusion (Sun et al., 2023). The first 3 minutes and last 2.5 minutes are excluded during preprocessing.
• Themes: Conversational prompts cover 161 topics; research analyses have focused on subsets such as “Crafts Hobbies,” “Day Time TV Shows,” “Summertime,” and “Halloween” (Sun et al., 2023, Alsuhaibani et al., 2023).
• Longitudinality: The design yields up to 158 themed sessions per participant, producing thousands of hours of home-recorded, naturalistic conversation (Alsuhaibani et al., 2023, Fard et al., 2024).

Transcripts

• Automatic Speech Recognition (ASR): 99.55% of session transcripts are generated by ASR systems, supplemented by a small set of manual transcripts for ASR refinement and quality control (Fard et al., 2024).
• Text structure: Transcripts are segmented into sentences or sentence-like units, serving as atomic linguistic features for NLP-based pipelines (Fard et al., 2024).

3. Preprocessing, Curation, and Labeling

Visual Data Processing

• Face localization: Frame-wise steps detect and retain only the subject’s primary face; methods use EasyOCR or CRAFT text detection for participant IDs, followed by RetinaFace detection and IoU-based selection to discard interviewer faces (Sun et al., 2023, Alsuhaibani et al., 2023).
• Quality assurance: Only video segments and frames rated “very good” or “good” are used for further analysis; false positive face crops are manually removed (Alsuhaibani et al., 2023).
• Standardization: Final cropped faces are resized to 96×96 pixels; variable original resolutions and down-sampled frame rates (target: 10 fps) ensure temporal consistency (Sun et al., 2023, Alsuhaibani et al., 2023).

Linguistic Data Processing

• Sentence segmentation: Raw ASR output, with no further spelling correction or filler deletion, is split into sentences; tokenization uses a pre-trained Sentence-Transformer model’s native tokenizer (Fard et al., 2024).
• Handling out-of-vocabulary: OOV words are mapped to [UNK]; disfluencies and punctuation retain default ASR handling (Fard et al., 2024).

Labeling

• Cognitive status: Video and transcript data are labeled as MCI or NC according to clinical diagnosis at baseline, with theme-level adjustments for follow-up timepoints (i.e., “Halloween” theme re-labeled at 6 months) (Alsuhaibani et al., 2023).
• Session- and segment-level aggregation: For video, segment-level predictions are aggregated by majority vote to produce session-level classifications (Sun et al., 2023, Alsuhaibani et al., 2023). For transcript sequences, average probabilities over all sequences per subject determine the final label (Fard et al., 2024).

4. Data Organization, Structure, and Imbalances

Modality	Constituents	Typical Counts / Structure
Video	30-min sessions × ≥4 per week	~24.5 min usable video per session (Sun et al., 2023)
Facial	Segments: T=16 (video), l=15 (CAE)	One segment = 16/15 consecutive face-frames
Transcript	Sentences per subject: 22–24	Sequences of γ=200 sentences (linguistic pipeline)

Inter- and Intra-class Imbalances

• Inter-class (“Hard/Easy” sample): Global class counts are slightly imbalanced (MCI/NC = 100/86, (Sun et al., 2023)); theme-level splits show greater variation (e.g., in “Day Time TV Shows,” MCI/NC = 20/21) (Sun et al., 2023).
• Intra-class (“Positive/Negative” sample): Some session segments/sequence windows from MCI subjects may display cognitively normal behaviors, increasing label noise. Video length and transcript quantity vary per subject (Sun et al., 2023, Fard et al., 2024).

5. Model Development, Loss Functions, and Evaluation Protocols

Visual Feature and Sequence Extraction

• Facial features: Convolutional autoencoders (CAE) based on truncated ResNet-50 extract 128-dimensional holistic encodings from 96×96 face crops (Alsuhaibani et al., 2023).
• Temporal modeling: Transformer encoders operate on sequences of CAE features with multi-level learned positional embeddings (frame, sequence, segment) (Alsuhaibani et al., 2023).
• Segment definitions: Segments are runs of frames containing the subject’s face, with sequences forming fixed-length overlapping or non-overlapping windows (default l = 15) (Alsuhaibani et al., 2023).

Linguistic Feature Extraction

• Sentence embeddings: Each sentence is embedded via a pre-trained “all-mpnet-base-v2” Sentence-Transformer; context modeling is handled by stacked Transformer blocks (Sentence Embedding + Sentence Cross Attention modules) (Fard et al., 2024).
• Sequence-level aggregation: Sequences of γ=200 consecutive sentences form input units for classification, with output labels averaged at the subject level (Fard et al., 2024).

Loss Functions for Imbalanced Data

Domain	Loss Function(s)	Targeted Imbalance
Vision (MC-ViViT) (Sun et al., 2023)	Focal Loss; AD-CORRE Loss; HP Loss (L_HP = L_FL + λ AD-CORRE(FD), λ=0.5)	Inter-class (“Hard/Easy”); Intra-class (“Positive/Negative”)
Linguistics (Fard et al., 2024)	InfoLoss (smoothing based on sequence count, KL-divergence)	Sequence-level label noise and subject-level imbalance

• Focal Loss: Emphasizes correct classification of “hard” samples (minority class), using class weighting and a focusing parameter γ (Sun et al., 2023).
• AD-CORRE Loss: Penalizes intra-class discordance using batch-variance, attention, and correlation matrices (Sun et al., 2023).
• InfoLoss: Smoothes one-hot sequence labels according to per-subject sequence frequency, increasing label entropy for over-represented subjects and reinforcing confidence for under-represented ones (Fard et al., 2024).

6. Empirical Results and Benchmarking

Vision-Based MCI Detection

• MC-ViViT: On “Crafts Hobbies” theme, video-level metrics are: 90.63% accuracy, F1 = 93.03%, AUC = 0.6042, sensitivity = 100%, specificity = 75%. Across four themes accuracy spans 85.37%–90.63%, F1 scores 85.00%–93.03%, AUC 0.4952–0.6122 (Sun et al., 2023).
• Facial Feature Transformer Model: Segment/sequence-based aggregation improves detection performance (up to 88% accuracy for theme “Halloween” with both segment and sequence embeddings; F1 = 0.89, AUC = 0.87) (Alsuhaibani et al., 2023). Ablation studies show 0% overlap between sequences and window size l=15 are empirically optimal.

Linguistic-Based MCI Detection

• Transformer-based NLP Pipeline: Subject-level mean accuracy = 85.16% (± 5.76), AUC = 84.75% (± 5.97) across five folds. Sequence-level metrics are lower: accuracy = 72.45%, AUC = 72.39% (Fard et al., 2024). InfoLoss raises the subject-level AUC by up to 8.4 percentage points over standard Cross-Entropy loss for subjects with fewer sequences.

Data Balance and Performance

• Class balance: Theme selection and rigorous participant selection protocols yield near-balanced MCI/NC splits in facial and linguistic subsets, minimizing confounding (Alsuhaibani et al., 2023, Fard et al., 2024).
• Imbalance effects: Weighted losses (Focal, InfoLoss) are necessary to maintain performance under unavoidable data nonuniformity.

7. Applications and Research Impact

• Automated screening: The I-CONECT dataset supports the development and benchmarking of scalable, vision-based and linguistic MCI detection systems deployable in non-clinical settings (Sun et al., 2023, Fard et al., 2024).
• Naturalistic behavioral analysis: The longitudinal, semi-structured, conversational structure enables the study of subtle verbal and non-verbal correlates of cognitive decline (e.g., micro-expressions, gaze, conversational spontaneity) not accessible via imaging or neuropsychological tests (Alsuhaibani et al., 2023).
• Ground-truth rigor: The clinical adjudication of MCI and NC labels ensures training and evaluation are robust to diagnostic confounding.
• Benchmarking: Multiple works now use the I-CONECT data as a common testbed for advanced deep learning architectures, novel loss functions, and cross-modal fusion approaches for aging-related neurocognitive analyses (Sun et al., 2023, Alsuhaibani et al., 2023, Fard et al., 2024).

A plausible implication is that future expansions of the I-CONECT protocol (e.g., increased sample size, broader linguistic and ethnic coverage, or multi-modal fusion) could accelerate the development of clinically actionable artificial intelligence for early dementia risk stratification.