---
title: 'CardioBench: Smartphone HR Testing Platform'
url: https://www.emergentmind.com/topics/cardiobench
type: topic
---

# CardioBench: Smartphone HR Testing Platform

CardioBench is a high-throughput bench-testing platform for smartphone applications that estimate heart rate from fingertip video photoplethysmography (PPG). It was introduced to address a specific validation problem in mobile health: smartphone-based heart-rate apps operate through the full camera and software stack of individual devices, yet the Android and iOS ecosystems are highly fragmented with respect to camera hardware, image signal processors, frame rates, operating systems, and other phone-specific behaviors. In that setting, manual device-by-device testing is slow, expensive, and incomplete, and the paper argues that, unlike pulse oximeters or wearables, there has not been a standardized functional tester for camera-based heart-rate apps despite their widespread use [2506.23414].

## 1. Motivation and problem domain

CardioBench is motivated by device fragmentation in smartphone-based heart-rate monitoring. Finger-over-camera PPG applications do not operate in an abstract algorithmic environment; they depend on the end-to-end behavior of the phone’s camera, timing, image pipeline, and operating system. The paper therefore frames validation as a systems problem rather than only an algorithm problem. Prior work had already shown substantial accuracy differences across devices and apps, but there was no standardized way to functionally test camera-based heart-rate software across a large device set before deployment [2506.23414].

The platform is positioned as a pre-deployment validation tool for rapidly probing performance, compatibility, and hidden failure modes. This emphasis is important because offline testing on prerecorded PPG databases cannot exercise the actual smartphone hardware pipeline. CardioBench instead tests the deployed acquisition-and-inference stack as it would run on real phones, while replacing the human finger-camera interaction with a controlled synthetic video input. This suggests a benchmarking philosophy centered on hardware-aware reproducibility rather than purely model-centric evaluation.

The platform also addresses scale. The paper explicitly treats manual serial testing as impractical, particularly when developers need to evaluate many device models, repeat tests across app releases, or check compatibility as new phones enter the market. CardioBench’s design therefore makes throughput a first-class property of the benchmark rather than a secondary implementation detail [2506.23414].

## 2. System architecture

CardioBench has three main components: a multi-phone test rig, a host controller, and a synthetic PPG video generation pipeline. The physical rig is an enclosed box with a monitor mounted on a swinging door and horizontal beams that can hold up to 12 smartphones at once, each facing the display. A grommeted side opening routes cables to the phones and simulator, and the modular design can be expanded with another monitor to increase throughput [2506.23414].

The host machine coordinates simultaneous playback of test videos on the monitor and automates device triggering and logging. In the validation study, the test application—Google Fit’s HR feature—was modified so that the host could initiate measurements and collect outputs automatically. That automation is central to CardioBench’s scalability, because it enables parallel rather than serial testing over multiple devices [2506.23414].

| Component | Description | Function |
|---|---|---|
| Multi-phone test rig | Enclosed box with monitor on a swinging door and beams for up to 12 smartphones | Parallel physical mounting and playback exposure |
| Host controller | Host machine coordinating playback, triggering, and logging | Synchronized automation of measurement runs |
| Synthetic video pipeline | Converts input PPG waveforms into RGB video sequences | Controlled generation of test stimuli |

This architecture makes CardioBench a bench tester in the literal sense: it is not only a dataset or evaluation script, but an integrated hardware-software apparatus for repeatable device-level measurement. A plausible implication is that its throughput advantage depends as much on orchestration and device control as on the realism of the synthetic PPG videos.

## 3. Synthetic PPG video generation

The synthetic video pipeline is the most novel part of CardioBench. The process begins with a desired PPG waveform, which may be simulated or drawn from real databases such as MIMIC/PhysioNet. The paper reports use of the NeuroKit PPG simulator because it supports control over heart rate, duration, respiratory sinus arrhythmia, baseline drift, motion artifacts, powerline artifacts, and similar properties [2506.23414].

The target PPG is downsampled to the video frame rate, inverted, and rescaled into RGB pixel-value ranges, yielding a mapped PPG waveform. This mapping step allows the platform to emulate different video-color distributions and signal strengths, including effects associated with skin pigmentation and lighting, using ranges informed by real finger-over-camera recordings from a prior clinical study. For each PPG sample, the system constructs an \(n \times m\) frame for each RGB channel by drawing random 8-bit integers from a Gaussian distribution centered on the desired floating-point RGB value. Stacking the three channels yields one color frame, and the sequence of frames becomes the synthetic test video [2506.23414].

In effect, CardioBench encodes physiological signal content into the temporal evolution of pixel color while preserving realistic frame-level variability. The paper emphasizes explicit control over both heart rate and signal quality. Synthetic videos were created at heart rates of 60, 80, 100, 120, and 180 bpm, and at four different PPG signal strengths representing different brightness and pulse amplitudes. Because the waveform source can also include arbitrary noise characteristics, the benchmark can span physiologically plausible extremes such as bradycardia and tachycardia, and can in principle represent abnormal rhythms such as atrial fibrillation if real waveforms are used as the source [2506.23414].

Signal quality is also parameterized. Drift, motion artifacts, powerline noise, and synthetic noise at desired signal-to-noise ratios can be added before RGB mapping. The resulting videos therefore act as controlled proxies for varying acquisition conditions. This suggests that CardioBench is designed not only for nominal accuracy measurement but also for systematic stress testing under perturbations that are difficult to stage reproducibly with human subjects.

## 4. Evaluation protocol and measurement criteria

CardioBench uses two main accuracy measures: mean absolute percentage error (MAPE) for heart-rate accuracy and Pearson correlation for signal fidelity. The heart-rate criterion follows the standard form

\[
\mathrm{MAPE} = \frac{1}{N}\sum_{i=1}^{N}\left|\frac{\hat{y}_i-y_i}{y_i}\right|\times 100\%,
\]

where \(y_i\) is the expected or input heart rate, \(\hat{y}_i\) is the measured heart rate from the app, and \(N\) is the number of paired observations. For waveform fidelity, the benchmark reports the Pearson correlation coefficient between the expected PPG signal embedded in the test video and the PPG signal recovered by the app. The paper also reports coefficients of variation (CoVs) across repeated runs to quantify reproducibility [2506.23414].

The benchmark’s device-level classification rule is operational rather than purely descriptive: a smartphone model is considered accurate if its MAPE is below 10%, matching ANSI/CTA standards for consumer HR monitors. That threshold converts continuous error measurement into a pass/fail compatibility judgment suitable for deployment screening [2506.23414].

Validation on a reference device was performed using a Pixel 3 running the clinically validated Google Fit HR app. Across 400 paired HR measurements from 20 videos tested over 20 rounds, the system achieved an overall MAPE of \(0.11\% \pm 0.001\%\) between the intended HR used to generate the videos and the HR measured by the phone, with Pearson correlation of 1.0 for HR agreement. At the waveform level, the correlation between the embedded PPG and the app-recorded PPG was \(0.92 \pm 0.008\). Reproducibility was also strong, with CoVs of 1.17% for MAPE and 0.90% for correlation over the 20 runs [2506.23414].

These results are presented as evidence that the synthetic video generator is highly faithful. More specifically, the reported waveform correlation indicates that the generated videos preserved rate, rhythm, and morphology well enough for the app to recover the signal rather than only the scalar heart-rate output.

## 5. Device classification and comparison with human-subject testing

After validation on the reference device, CardioBench was used to classify 20 smartphone models from 7 manufacturers. All 20 models met the ANSI/CTA accuracy threshold in bench testing, with MAPEs ranging from 0.11% to 5.19%, all below the 10% cutoff. The worst bench result was the LG Nexus 5X at 5.19%, and its errors were concentrated at high heart rates and associated with frequent frame drops, which corrupted the recovered PPG [2506.23414].

That failure mode is significant because it illustrates the kind of device-specific issue the benchmark is meant to expose. The problem was not described as an abstract algorithmic weakness; it was linked to hardware-software timing behavior visible only when the app was exercised on the actual smartphone pipeline. This suggests that CardioBench is especially valuable for identifying incompatibilities related to camera timing, image signal processor configuration, unstable frame rates, or poor signal acquisition.

The paper then compares bench-testing outcomes with a prospective clinical study involving human participants. In that study, 74 participants were analyzed after 6 were excluded for missing reference pulse oximeter data. Participants were diverse in age and skin tone, each completed 40 HR measurements with a mix of rest and post-exercise recordings, and an FDA-cleared pulse oximeter served as the reference. Clinical MAPEs were slightly higher than bench-test MAPEs, ranging from 1.74% to 5.11%, and there were more high-error outliers. Nevertheless, every device still met the ANSI/CTA standard of MAPE <10%, so CardioBench correctly classified all 20 phones as accurate, yielding a high positive predictive value [2506.23414].

The comparison also revealed asymmetries between bench and clinical settings. The LG Nexus 5X performed worse in bench testing than in the clinical study, likely because the clinical recordings did not reach the highest heart-rate values where the device failed in bench mode. Conversely, the Galaxy Note 20 Ultra showed the largest clinical MAPE, 5.11%, because two large outliers coincided with phone motion, supported by accelerometer evidence. These cases clarify that bench testing and human-subject testing are not interchangeable; rather, they probe different parts of the error surface.

## 6. Scope, limitations, and broader use of the name

CardioBench is presented as a standardized, repeatable way to test camera-based heart-rate apps before deployment across many phone models and across controllable physiological conditions. The paper emphasizes practical uses such as rapid iteration, regression testing between app releases, and compatibility checks when new devices are introduced. Its principal advantage over offline algorithm testing is that it exercises the actual smartphone hardware pipeline rather than only signal-processing logic on prerecorded data [2506.23414].

The platform also has explicit limitations. It cannot simulate the full finger-camera mechanical interface or capture all human-factor effects such as ergonomics, finger pressure, or flashlight-specific issues. The paper therefore does not treat bench testing as a replacement for all clinical or usability validation. Instead, it is positioned as a scalable pre-deployment filter that can reduce reliance on expensive, slow human-subject testing while improving confidence in device compatibility and app performance [2506.23414].

The term “CardioBench” is not unique in the recent cardiovascular AI literature. It has also been used for a downstream utility benchmark for PPG-to-ECG translation [2308.13568], a comprehensive benchmark for ECG time-series analysis [2507.14206], and a public benchmark for echocardiography foundation models [2510.00520]. In the context of smartphone video heart-rate measurement, however, CardioBench refers specifically to the high-throughput bench-testing platform for finger-over-camera PPG apps introduced in “A High-Throughput Platform to Bench Test Smartphone-Based Heart Rate Measurements Derived From Video” [2506.23414].

Source: https://www.emergentmind.com/topics/cardiobench