---
title: 'I-CONECT Dataset: Multi-Modal MCI Analysis'
url: https://www.emergentmind.com/topics/i-conect-dataset
type: topic
---

# I-CONECT Dataset: Multi-Modal MCI Analysis

The I-CONECT (Internet-Based Conversational Engagement Clinical Trial) dataset is a suite of large-scale, multi-modal resources created to enable automated analysis of mild cognitive impairment (MCI) in older adults. Originating from a randomized controlled trial (ClinicalTrials.gov NCT02871921) aimed at enhancing cognitive reserve in socially isolated adults aged 75 or older via frequent semi-structured video chat, the dataset includes longitudinal video, audio, and transcript modalities. I-CONECT is distinguished by its high session frequency (four video interviews per week over six months), naturalistic conversational context, and rigorous ground-truth cognitive labels. The dataset has served as a benchmark for deep learning research in vision-based, linguistic, and multi-modal MCI detection.

## 1. Composition and Participant Demographics

### Cognitive and Demographic Characteristics

- **Participant sets**: The main cohort includes 186 older adults (≥75 years) recruited in Portland, Oregon and Detroit, Michigan: 100 diagnosed with MCI and 86 with normal cognition (NC) [2304.05292]. Several subcohorts appear in domain-specific studies, e.g., 68 subjects (34 MCI, 34 NC) for linguistic analysis [2402.01690], and 69 participants for facial features research [2308.15624].
- **Gender balance**: Modest male–female variation across conversation themes; for example, “Crafts Hobbies” (n = 32 videos) includes 11 males and 21 females (20 MCI, 12 NC) [2304.05292]. Theme-level splits for other subsets are closely balanced.
- **Age range**: All subjects are ≥75; mean ages per theme typically 80–81 years [2308.15624].
- **Educational background**: Means range from 15.3 to 15.8 years (per theme, SD ≈ 2.5) [2308.15624].
- **Session protocol**: Each subject participated in 30-minute video chat sessions four times per week over six months, each regarding one of 161 daily-life themes (e.g., “Summertime,” “Military Service”) [2308.15624, 2402.01690].

## 2. Data Modalities and Collection Protocol

### Video and Audio Recordings

- **Recording configuration**: Sessions were conducted using study-provided tablets or webcams deployed in participants’ homes, with unconstrained environmental conditions (variable lighting, backgrounds, camera angles, and occlusion) [2308.15624].
- **Session structure**: Each video records a two-participant split-screen interaction, typically ~30 minutes after removal of preamble/conclusion [2304.05292]. The first 3 minutes and last 2.5 minutes are excluded during preprocessing.
- **Themes**: Conversational prompts cover 161 topics; research analyses have focused on subsets such as “Crafts Hobbies,” “Day Time TV Shows,” “Summertime,” and “Halloween” [2304.05292, 2308.15624].
- **Longitudinality**: The design yields up to 158 themed sessions per participant, producing thousands of hours of home-recorded, naturalistic conversation [2308.15624, 2402.01690].

### Transcripts

- **Automatic Speech Recognition (ASR)**: 99.55% of session transcripts are generated by ASR systems, supplemented by a small set of manual transcripts for ASR refinement and quality control [2402.01690].
- **Text structure**: Transcripts are segmented into sentences or sentence-like units, serving as atomic linguistic features for NLP-based pipelines [2402.01690].

## 3. Preprocessing, Curation, and Labeling

### Visual Data Processing

- **Face localization**: Frame-wise steps detect and retain only the subject’s primary face; methods use EasyOCR or CRAFT text detection for participant IDs, followed by RetinaFace detection and IoU-based selection to discard interviewer faces [2304.05292, 2308.15624].
- **Quality assurance**: Only video segments and frames rated “very good” or “good” are used for further analysis; false positive face crops are manually removed [2308.15624].
- **Standardization**: Final cropped faces are resized to 96×96 pixels; variable original resolutions and down-sampled frame rates (target: 10 fps) ensure temporal consistency [2304.05292, 2308.15624].

### Linguistic Data Processing

- **Sentence segmentation**: Raw ASR output, with no further spelling correction or filler deletion, is split into sentences; tokenization uses a pre-trained Sentence-Transformer model’s native tokenizer [2402.01690].
- **Handling out-of-vocabulary**: OOV words are mapped to [UNK]; disfluencies and punctuation retain default ASR handling [2402.01690].

### Labeling

- **Cognitive status**: Video and transcript data are labeled as MCI or NC according to clinical diagnosis at baseline, with theme-level adjustments for follow-up timepoints (i.e., “Halloween” theme re-labeled at 6 months) [2308.15624].
- **Session- and segment-level aggregation**: For video, segment-level predictions are aggregated by majority vote to produce session-level classifications [2304.05292, 2308.15624]. For transcript sequences, average probabilities over all sequences per subject determine the final label [2402.01690].

## 4. Data Organization, Structure, and Imbalances

| Modality    | Constituents                         | Typical Counts / Structure                                  |
|-------------|--------------------------------------|-------------------------------------------------------------|
| Video       | 30-min sessions × ≥4 per week        | ~24.5 min usable video per session [2304.05292]             |
| Facial      | Segments: T=16 (video), l=15 (CAE)   | One segment = 16/15 consecutive face-frames                 |
| Transcript  | Sentences per subject: 22–24         | Sequences of γ=200 sentences (linguistic pipeline)          |

### Inter- and Intra-class Imbalances

- **Inter-class (“Hard/Easy” sample)**: Global class counts are slightly imbalanced (MCI/NC = 100/86, [2304.05292]); theme-level splits show greater variation (e.g., in “Day Time TV Shows,” MCI/NC = 20/21) [2304.05292].
- **Intra-class (“Positive/Negative” sample)**: Some session segments/sequence windows from MCI subjects may display cognitively normal behaviors, increasing label noise. Video length and transcript quantity vary per subject [2304.05292, 2402.01690].

## 5. Model Development, Loss Functions, and Evaluation Protocols

### Visual Feature and Sequence Extraction

- **Facial features**: Convolutional autoencoders (CAE) based on truncated ResNet-50 extract 128-dimensional holistic encodings from 96×96 face crops [2308.15624].
- **Temporal modeling**: Transformer encoders operate on sequences of CAE features with multi-level learned positional embeddings (frame, sequence, segment) [2308.15624].
- **Segment definitions**: Segments are runs of frames containing the subject’s face, with sequences forming fixed-length overlapping or non-overlapping windows (default l = 15) [2308.15624].

### Linguistic Feature Extraction

- **Sentence embeddings**: Each sentence is embedded via a pre-trained “all-mpnet-base-v2” Sentence-Transformer; context modeling is handled by stacked Transformer blocks (Sentence Embedding + Sentence Cross Attention modules) [2402.01690].
- **Sequence-level aggregation**: Sequences of γ=200 consecutive sentences form input units for classification, with output labels averaged at the subject level [2402.01690].

### Loss Functions for Imbalanced Data

| Domain          | Loss Function(s)                             | Targeted Imbalance                                         |
|-----------------|---------------------------------------------|------------------------------------------------------------|
| Vision (MC-ViViT) [2304.05292] | Focal Loss; AD-CORRE Loss; HP Loss (L_HP = L_FL + λ AD-CORRE(FD), λ=0.5) | Inter-class (“Hard/Easy”); Intra-class (“Positive/Negative”) |
| Linguistics [2402.01690]        | InfoLoss (smoothing based on sequence count, KL-divergence)      | Sequence-level label noise and subject-level imbalance      |

- **Focal Loss**: Emphasizes correct classification of “hard” samples (minority class), using class weighting and a focusing parameter γ [2304.05292].
- **AD-CORRE Loss**: Penalizes intra-class discordance using batch-variance, attention, and correlation matrices [2304.05292].
- **InfoLoss**: Smoothes one-hot sequence labels according to per-subject sequence frequency, increasing label entropy for over-represented subjects and reinforcing confidence for under-represented ones [2402.01690].

## 6. Empirical Results and Benchmarking

### Vision-Based MCI Detection

- **MC-ViViT**: On “Crafts Hobbies” theme, video-level metrics are: 90.63% accuracy, F1 = 93.03%, AUC = 0.6042, sensitivity = 100%, specificity = 75%. Across four themes accuracy spans 85.37%–90.63%, F1 scores 85.00%–93.03%, AUC 0.4952–0.6122 [2304.05292].
- **Facial Feature Transformer Model**: Segment/sequence-based aggregation improves detection performance (up to 88% accuracy for theme “Halloween” with both segment and sequence embeddings; F1 = 0.89, AUC = 0.87) [2308.15624]. Ablation studies show 0% overlap between sequences and window size l=15 are empirically optimal.

### Linguistic-Based MCI Detection

- **Transformer-based NLP Pipeline**: Subject-level mean accuracy = 85.16% (± 5.76), AUC = 84.75% (± 5.97) across five folds. Sequence-level metrics are lower: accuracy = 72.45%, AUC = 72.39% [2402.01690]. InfoLoss raises the subject-level AUC by up to 8.4 percentage points over standard Cross-Entropy loss for subjects with fewer sequences.

### Data Balance and Performance

- **Class balance**: Theme selection and rigorous participant selection protocols yield near-balanced MCI/NC splits in facial and linguistic subsets, minimizing confounding [2308.15624, 2402.01690].
- **Imbalance effects**: Weighted losses (Focal, InfoLoss) are necessary to maintain performance under unavoidable data nonuniformity.

## 7. Applications and Research Impact

- **Automated screening**: The I-CONECT dataset supports the development and benchmarking of scalable, vision-based and linguistic MCI detection systems deployable in non-clinical settings [2304.05292, 2402.01690].
- **Naturalistic behavioral analysis**: The longitudinal, semi-structured, conversational structure enables the study of subtle verbal and non-verbal correlates of cognitive decline (e.g., micro-expressions, gaze, conversational spontaneity) not accessible via imaging or neuropsychological tests [2308.15624].
- **Ground-truth rigor**: The clinical adjudication of MCI and NC labels ensures training and evaluation are robust to diagnostic confounding.
- **Benchmarking**: Multiple works now use the I-CONECT data as a common testbed for advanced deep learning architectures, novel loss functions, and cross-modal fusion approaches for aging-related neurocognitive analyses [2304.05292, 2308.15624, 2402.01690].

A plausible implication is that future expansions of the I-CONECT protocol (e.g., increased sample size, broader linguistic and ethnic coverage, or multi-modal fusion) could accelerate the development of clinically actionable artificial intelligence for early dementia risk stratification.

Source: https://www.emergentmind.com/topics/i-conect-dataset