GPT-PPG: A GPT-based Foundation Model for Photoplethysmography Signals (2503.08015v1)

Abstract: This study introduces a novel application of a Generative Pre-trained Transformer (GPT) model tailored for photoplethysmography (PPG) signals, serving as a foundation model for various downstream tasks. Adapting the standard GPT architecture to suit the continuous characteristics of PPG signals, our approach demonstrates promising results. Our models are pre-trained on our extensive dataset that contains more than 200 million 30s PPG samples. We explored different supervised fine-tuning techniques to adapt our model to downstream tasks, resulting in performance comparable to or surpassing current state-of-the-art (SOTA) methods in tasks like atrial fibrillation detection. A standout feature of our GPT model is its inherent capability to perform generative tasks such as signal denoising effectively, without the need for further fine-tuning. This success is attributed to the generative nature of the GPT framework.

• The paper introduces GPT-PPG, which adapts the GPT architecture to analyze continuous PPG signals for heart rate estimation and atrial fibrillation detection.
• It employs a mixed-objective fine-tuning framework that combines signal modeling with task-specific objectives to enhance prediction accuracy.
• The approach demonstrates strong generative capabilities for signal denoising on large clinical datasets, though cross-domain generalization remains a challenge.

GPT-PPG: A GPT-based Foundation Model for Photoplethysmography Signals

The paper "GPT-PPG: A GPT-based Foundation Model for Photoplethysmography Signals" (2503.08015) introduces GPT-PPG, a novel foundation model for photoplethysmography (PPG) signals. Leveraging the architecture of Generative Pre-trained Transformers (GPT), the paper adapts this model to the continuous nature of PPG data for various downstream tasks, including heart rate estimation and atrial fibrillation detection. The paper highlights the adaptation of the GPT architecture to address the challenges posed by the continuous characteristics of PPG signals, utilizing a comprehensive dataset of over 200 million 30-second PPG samples for pre-training. The research provides an extensive analysis of GPT-PPG's performance across a range of tasks, underscoring its generative capabilities and the robustness of its feature extraction, even in out-of-domain (OOD) scenarios.

Data Preparation and Pre-training Considerations

The dataset for GPT-PPG pre-training was derived from UCSF Medical Center's patient monitoring systems, encompassing data from over 24,100 patients. With a total of 2.6 million hours of continuous signal recordings, the data included ECG, PPG, and respiration rate signals from patients monitored between March 2013 and December 2018. The PPG signals were segmented into 30-second, 40Hz-recorded strips and normalized to fit the interval $[0,1]$ via min-max normalization, ensuring that no assumptions about data point distribution were made. This preprocessing step was pivotal in facilitating the application of the GPT architecture to continuous physiological data typically characterized by noise and variability.

Figure 1: Average Signal Modeling Loss. The datasets that are boxed in red are all heart rate estimation tasks. It is clear that on those datasets, our model is less competitive against the baselines compared to our model's performance on other datasets.

For the GPT pre-training phase, four distinct models were trained with varying parameters, from 19 million to 1 billion. This variation allowed the authors to explore the effects of model scaling on performance. GPT-PPG employs a unique approach to predictor architecture and loss function by adapting the traditionally discrete, classification-focused GPT to predict continuous time-series data through a logit-Laplace distribution loss function, avoiding model collapse issues encountered with using simple MSE (Equation 1).

Mixed Objective Fine-tuning Framework

The paper posits a mixed-objective fine-tuning framework that ingeniously combines signal modeling with task-specific objectives. The framework's adaptability to different sequence lengths and the ability to leverage attention pooling mechanisms for feature aggregation is a notable advantage of the GPT architecture. Attention pooling facilitates robust feature extraction, while the model's generating capabilities are retained for seamless adjustment to various PPG datasets and tasks.

Figure 2: Mixed-objective Fine-tuning Framework

One method explored is the Bidirectional Feature Extraction technique, which adapts the GPT model to use bidirectional attention for augmented prediction accuracy. As demonstrated in Table 4, this approach significantly improves performance across several downstream tasks, notably heart rate estimation.

Parameter-Efficient Fine-tuning

The computational demands of fine-tuning extensive GPT models can be substantial, particularly at higher parameter scales. The paper examines more efficient strategies for fine-tuning while preserving accuracy. It leverages the fallback method, an innovative technique using likelihood-informed predictions to enhance performance without full model retraining. As seen in Table 5, this method proves effective, especially for datasets with distribution discrepancies from the pre-training data, showcasing robust improvement in prediction performance with minimal additional computational burden.

Generative Capabilities and Signal Denoising

One of the GPT-PPG's salient features is its effective handling of generative tasks like signal denoising. The authors provide qualitative and quantitative evidence of the model's proficiency at reconstructing missing portions of PPG signals, even when a significant percentage is masked (Figures 3, 5). This attribute is particularly beneficial for applications in noisy clinical environments and can potentially extend to other signal types.

Figure 4: AF Detection

Figure 5: Scaling Curve of GPT-PPG. The error bars are dropped for heart rate estimation tasks for readability (the error bars are on the scale of the mean, like in respiration rate estimation task). We observe much smaller error bars for AF detection and BP estimation, since the error bars in those two datasets are obtained by training and testing the model on the same train/test split multiple times. The variance comes from the inherent randomness of the training process, unlike in RR and HR where the train/test split differ in each trial. The error bar for BP is especially small potentially due to the large size of the dataset.

\section{Results and Limitations}

The GPT-PPG demonstrated robust performance in AF detection and blood pressure estimation, outperforming baseline models across various subsets of PPG data (Table 3). However, the model showed limitations in cross-domain generalization, particularly in heart rate tasks with distribution mismatch. The scaling analysis reveals that while increasing model size enhances performance, the most significant improvements occur from the leap to the 85M model, indicating a balance between size and computational efficiency (Figure 5).

The limitations of the model include challenges in generalizing to datasets with divergent distributions and the substantial computational demands associated with fine-tuning large foundation models like GPT-PPG. While the model performs well in aligned datasets, its efficacy diminishes on datasets with significant distribution differences from its pre-training set (Table 5).

Concluding Remarks

This essay provides an overview of "GPT-PPG: A GPT-based Foundation Model for Photoplethysmography Signals" (2503.08015), a paper that leverages the power of generative pre-training transformers for the analysis of PPG signals. By adapting the architecture to suit continuous time-series data, the paper demonstrates robust downstream performance, effective generative capabilities, and promising prospects for enhancing biosignal analysis through fine-tuning strategies. However, it identifies the need for further research to improve OOD generalization and efficiency in deployment. Future work may address the exploration of additional modalities and advanced fine-tuning strategies, unlocking new opportunities for foundation models in healthcare applications.