PixleepFlow: Deep Learning Sensor Fusion

• The paper introduces PixleepFlow, a deep learning framework that converts multi-channel lifelog sensor data into composite images for predicting sleep quality and stress.
• It details a robust pipeline for resampling, normalization, and barcode-like image construction that enhances CNN-based predictions compared to traditional raw data methods.
• Experimental results show that image-based inputs achieve up to a 0.746 macro-F1 score, with Explainable AI techniques enabling clear sensor attribution.

PixleepFlow is a deep learning-based framework that transforms heterogeneous, time-synchronized lifelog sensor data into composite images, enabling the prediction of sleep quality and stress levels using convolutional neural networks (CNNs) and providing model interpretability through Explainable AI techniques. Developed in the context of cross-modal health monitoring, PixleepFlow leverages pixel-wise representations of multi-channel sensor time series as the input modality for supervised prediction of daily well-being metrics, with performance surpassing traditional raw or frequency-domain feature pipelines (Na et al., 13 Feb 2025).

1. System Architecture and Image-Based Data Representation

PixleepFlow receives multi-channel time-series data from wearable and smartphone sensors—including accelerometry, heart rate, activity classification, and optionally GPS, ambient light, and step counts—spanning a full 24-hour period at resolutions up to 1 Hz (86 400 time steps per channel). The end-to-end data processing pipeline comprises:

1. Resampling and Synchronization: All sensor streams are resampled to a unified temporal grid (1 Hz) using nearest-neighbor selection, ensuring precise alignment across modalities.
2. Normalization and Gap Filling: Each channel undergoes per-day min-max normalization. Intra-day missing intervals are addressed with linear interpolation; no extrapolation occurs outside observed ranges.
3. Composite Image Construction: The normalized time series for each channel is tiled as a contiguous horizontal band (“barcode” encoding) within a 3-channel image $I \in \mathbb{R}^{H \times W \times 3}$, where the vertical coordinate codes the channel index and the horizontal axis encodes time. A color mapping (e.g., “jet” from Matplotlib) visually encodes normalized values (Section II.D of (Na et al., 13 Feb 2025)).
4. Downsampling: High-frequency vectors (length 86 400) are collapsed to the target image width (e.g., $W=224$) via averaging.

The resulting images encapsulate an entire day of multivariate behavioral data into a format amenable to convolutional analysis.

2. Backbone CNN Models and Supervised Training

PixleepFlow utilizes a 2D convolutional architecture, specifically SE-ResNeXt101_32×4d or ResNeXt101_32×32d, pretrained on standard image datasets and fine-tuned on the composite lifelog representations. The model's architecture consists of:

• SE-ResNeXt backbone with grouped convolutions (cardinality 32, bottleneck width 4)
• Global average pooling followed by a dropout (rate 0.5)
• Fully connected output projecting to seven binary targets (sleep quality Q1–Q3, stress S1–S4)
• Sigmoid activations for multi-label prediction

Optimization is performed using AdamW (weight decay 0.1, initial learning rate $5 \times 10^{-5}$, cosine annealing over 200 epochs, early stopping at epoch 100 if loss exceeds 0.8). Training employs K-fold cross-validation (K=5), with stratified folds by subject and final predictions computed as an ensemble average (Na et al., 13 Feb 2025).

The multi-label binary cross-entropy loss for targets $y \in \{0,1\}^7$ and predictions $\hat{y}$ is:

$$L(\hat{y}, y) = -\sum_{i=1}^7 [y_i \log \hat{y}_i + (1-y_i) \log(1-\hat{y}_i)]$$

3. Sensor Modalities, Input Types, and Performance Analysis

PixleepFlow investigates various channel combinations and input modalities:

• 5-channel set: Accelerometer (x/y/z), heart rate, activity ($F_1$(macro): 0.680 for image-based, 0.636 for raw, 0.613 for spectrogram)
• 11-channel set: Adds GPS (lat/lon/alt/speed), ambient light, and step count ($F_1$(macro): 0.746 for image-based, 0.699 for raw, 0.637 for spectrogram)

Channel Set	Input Type	Macro-$F_1$
5	Image	0.680
5	Raw	0.636
5	Spectrogram	0.613
11	Image	0.746
11	Raw	0.699
11	Spectrogram	0.637

On a per-label basis, the 11-channel configuration boosts $F_1$ for difficult classes (e.g., Q3: 0.308 → 0.767). Key discriminative signal for sleep and stress comes from the Y/Z axes of the accelerometer and heart rate, whereas activity channels provide marginal contributions (Na et al., 13 Feb 2025).

Experiments on extended datasets with 7 and 18 channels indicate that 5-channel, full-rate (1 Hz) data offers optimal generalization, whereas denser channel sets benefit from down-sampling (1/60 Hz functions as regularization). The addition of ambient sound embeddings further aids sleep-related label prediction.

4. Explainable AI: Full-CAM for Sensor Attribution

Model interpretability is addressed using the Full-Class Activation Map (Full-CAM) technique, a gradient-based approach that highlights contributions of individual image regions—and, by construction, sensor channels and times—to each predicted label:

$W=224$0

where $W=224$1 is the logit for class $W=224$2, $W=224$3 are the final activation maps, and $W=224$4 are intermediate biases.

Overlaying Full-CAMs on composite images demonstrates that classifier attention consistently localizes to bands corresponding to the Y/Z axes of accelerometers and heart-rate channels, with low responses to activity-modality bands. This mapping provides actionable interpretability for both researchers and end-users (Na et al., 13 Feb 2025).

5. Dataset, Experimental Setup, and Results

PixleepFlow is validated on lifelog data from eight subjects (220 days, 4/4 split train/test). Across all experiments, the image-based approach outperforms raw and spectrogram input formats by 4–6 macro-$W=224$5 points. Key tabled results:

• With 5 channels: Mean macro-$W=224$6 = 0.680 (image), 0.636 (raw), 0.613 (spectrogram)
• With 11 channels: Mean macro-$W=224$7 = 0.746 (image), 0.699 (raw), 0.637 (spectrogram)
• Sleep-duration prediction ($W=224$8: ≈0.60) remains the most challenging metric

An extended analysis using additional channels, sound embeddings, and optimization schedules confirms the robustness of PixleepFlow to input modality and system configuration, with early stopping and sharpness-aware minimization (SAM) deployed to mitigate overfitting (Na et al., 13 Feb 2025).

6. Limitations and Prospects for Extension

The small cohort size and train/test split (n=8, binary split) introduces potential for overfitting, despite regularization techniques. Implementation is currently limited to offline batch inference; real-time, on-device deployment is suggested as a future direction. Additional improvements may be achievable by integrating more diverse sensor modalities (e.g., thermal, EEG), refining image encoding procedures (learned tiling or attention-based warping), and scaling to larger, heterogeneous populations (Na et al., 13 Feb 2025).

A plausible implication is that PixleepFlow's interpretable, image-based fusion of wearable signals constitutes a transferable paradigm for biomedical time-series, supporting both end-user understanding and high-accuracy supervised prediction of health-related status markers.