MultiPhysio-HRC: Physiology-Aware HRC

• MultiPhysio-HRC is a multimodal physiological monitoring dataset integrating biosignals, audio, and facial data to profile human-robot collaboration.
• It employs synchronized acquisition through LSL and robust preprocessing pipelines to ensure sub-millisecond precision and effective artifact removal.
• The dataset supports diverse experimental protocols and benchmark models for assessing stress, cognitive load, and operator performance in HRC.

Searching arXiv for MultiPhysio-HRC and closely related HRC physiological monitoring work. arXiv search query: "MultiPhysio-HRC human robot collaboration physiological monitoring" Searching arXiv API for the provided paper ids and related titles. $V_2$6 $V_2$7 MultiPhysio-HRC denotes a multimodal physiological monitoring and data resource for human-robot collaboration (HRC) that combines synchronized biosignals, audio, and facial information with task structure and post-task psychological annotations to characterize human psycho-physical state during collaborative work (Bussolan et al., 1 Oct 2025). Its lineage is tied to earlier HRC monitoring work that formalized event-marker generation, continuous synchronized acquisition, and post-hoc visualization during robot-mediated stimuli (Savur et al., 2019), and to the HRC-SoS experimentation platform, which framed HRC as a system of systems organized around Awareness, Intelligence and Compliance and explicitly highlighted a subsystem for monitoring human physiological feedback during collaboration (Savur et al., 2019). In that sense, MultiPhysio-HRC is not only a dataset name but part of a broader methodological trajectory in physiology-aware collaborative robotics.

1. Origins and conceptual position

The immediate conceptual background is the HRC-SoS platform, which presents an experimentation platform for human robot collaboration as a system of systems and proposes a conceptual framework describing the aspects of Human Robot Collaboration as Awareness, Intelligence and Compliance (Savur et al., 2019). The platform discussion centers on subsystems including the digital twin, motion capture system, human-physiological monitoring system, data collection system, and robot control and interface systems. A highlighted subsystem is the one with the ability to monitor human physiological feedback during a human robot collaboration task.

The subsequent monitoring framework organizes physiological HRC experimentation into five primary modules plus a real-time communication and synchronization backbone: Physical-World Sensor Acquisition, Event Marker Generator, Synchronization Engine, Data Storage, and Visualization & Analysis (Savur et al., 2019). In that framework, each acquisition node writes into an LSL outlet, the Event Marker Generator writes markers into LSL, the Synchronization Engine arbitrates time and fans out to recorders, and the Visualizer subscribes to LSL or ROS topics via an lsl2ros bridge.

This progression suggests that MultiPhysio-HRC should be understood as the consolidation of two concerns that were initially treated as infrastructural requirements: first, end-to-end synchronized acquisition of physiological responses during HRC; second, reproducible organization of those signals into a reusable corpus. The 2025 release makes that consolidation explicit by introducing MultiPhysio-HRC as a public multimodal dataset for industrial HRC (Bussolan et al., 1 Oct 2025).

2. Experimental scope and task protocols

MultiPhysio-HRC adopts a two-day protocol with distinct elicitation regimes (Bussolan et al., 1 Oct 2025). Day 1 begins with 2 min rest, followed by five cognitive or VR tasks in randomized order, with questionnaires after each. The controlled cognitive tasks are the Stroop Color-Word Test, N-Back, Mental Arithmetic, Tower of Hanoi, and a guided breathing exercise. During all but Hanoi and breathing, a ticking clock and error buzzer induced time pressure. The Day 1 protocol also includes the immersive VR task “Richie’s Plank Experience” to elicit high arousal or fear. Day 2 begins with 5 min rest and then alternates between manual e-bike battery disassembly and collaborative disassembly, where a Fanuc CRX-20 cobot assists via a voice-command interface and HTN planner. Manual and HRC disassembly are each repeated up to five times to induce fatigue.

The participant pool comprises $N = 55$ on Day 1, with $N = 42$ returning on Day 2; the reported age is $27.98 \pm 10.22$ years, with 48 male and 7 female participants drawn from engineering students, researchers, and professionals (Bussolan et al., 1 Oct 2025). Ground-truth questionnaires—STAI-Y1, NASA-TLX, and SAM—were administered immediately after every task block so that labels were synchronized with physiological recordings.

Earlier HRC monitoring studies used more stimulus-centric robot protocols rather than the two-day multimodal corpus design (Savur et al., 2019). In Case Study I, a UR5e loading/unloading task used four subtasks defined by $\{\text{Normal, High}\}$ acceleration $\times \{\text{Fixed, Random}\}$ trajectory. In Case Study II, a UR10 assembly scenario used speed-and-separation monitoring with three safety modes: Normal, Reduced, and Stop. These earlier case studies are important because they establish the logic later retained by MultiPhysio-HRC: robot motion, task structure, and safety state are treated as experimentally meaningful stimuli rather than mere context.

3. Instrumentation, modalities, and synchronization

The published MultiPhysio-HRC dataset includes EEG, ECG, EDA, RESP, EMG, voice recordings, and facial action units (Bussolan et al., 1 Oct 2025). EEG is recorded with the Bitbrain Diadem dry EEG device using 12 electrodes at AF7, Fp1, Fp2, AF8, F3, F4, P3, P4, PO7, O1, O2, and PO8, plus earlobe ground and reference, at 256 Hz. ECG, EDA, RESP, and EMG are acquired with the Bitbrain Versatile Bio system, all sampled at 256 Hz and synchronized via SennsLab. EDA is measured on the fingers of the non-dominant hand, RESP via a chest belt, and EMG on the right trapezius. Audio is collected through a Bluetooth lapel microphone as raw .wav, and video is collected at 30 fps via webcam as .mp4 or raw frames, with synchronization metadata generated by SennsLab.

The earlier framework defined a broader acquisition envelope that also included GSR, PPG, respiration, motion-capture, gaze camera, and an eye-tracker providing pupil diameter at 120 Hz (Savur et al., 2019). It specifies ECG at 512 Hz and 16 bit, GSR at 32 Hz and 16 bit, EMG at 1 kHz and 12 bit, EEG with 8–32 channels at 256 Hz and 24 bit, PPG at 128 Hz and 16 bit, and a respiration belt at 64 Hz and 16 bit. The HRC-SoS physiological subsystem similarly emphasizes multimodal wearable sensing, including EEG, ECG, EMG, EDA, pupil dilation and gaze position, and a tri-axial accelerometer, with all streams exposed as discoverable LSL streams such as “BITalino_ECG” or “PupilGaze” (Savur et al., 2019).

Synchronization is a defining architectural element across this line of work. In the framework, Lab Streaming Layer provides sub-millisecond timestamps and clock synchronization, ROS-Bag and LabRecorder record all streams for post analysis, and event markers are injected into LSL as a dedicated stream (Savur et al., 2019). In HRC-SoS, LSL provides sub-millisecond clock synchronization across physiological devices, OptiTrack, and the robot controller; robot events and motion-capture skeletal poses are also streamed into LSL, and downstream processing merges streams by their LSL timestamps with no post-hoc realignment required (Savur et al., 2019). This infrastructure is the technical basis for relating transient physiological responses to robot actions, environmental changes, and task epochs.

4. Signal processing and feature construction

The signal-processing stack is explicitly multimodal. In the 2019 framework, per-channel preprocessing includes band-pass filtering, a notch filter at 50/60 Hz for line noise, artifact removal, windowing, and z-score normalization computed per-channel per-task (Savur et al., 2019). EEG uses ICA-based removal of ocular and muscle components, ECG uses peak-based rejection if $|\Delta RR| > 300\,\mathrm{ms}$, and spectral estimates are formed with a Hamming window,

$$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$

In the 2025 dataset, EEG preprocessing uses a 0.5–40 Hz band-pass and a 49–51 Hz notch; ECG uses a $V_2$-lead with 0.05–40 Hz band-pass plus Savitzky-Golay smoothing; EDA uses low-pass 10 Hz, smoothing, down-sampling to 100 Hz, and phasic/tonic decomposition through cvxEDA; RESP uses 0.03–5 Hz band-pass; and EMG uses a 10–500 Hz band-pass with linear detrending according to SENIAM (Bussolan et al., 1 Oct 2025). Feature windows are 60 s non-overlapping for ECG, EDA, EMG, and RESP, and 5 s with 50% overlap for EEG.

Feature extraction spans standard autonomic, muscular, respiratory, electrophysiological, facial, and vocal descriptors (Bussolan et al., 1 Oct 2025). ECG features include time- and frequency-domain HRV metrics such as SDNN, RMSSD, pNN50, and LF/HF ratio. The reported RMSSD definition is

$$\mathrm{RMSSD} = \sqrt{\frac{1}{N-1}\sum_{i=1}^{N-1} (RR_{i+1} - RR_i)^2}\,.$$

EDA features include phasic event count, mean SCR amplitude, and tonic level. EMG features include mean absolute value, zero-crossings, and waveform length. RESP features include breathing rate and spectral power. EEG features include band powers via Welch’s PSD over $\delta(1\text{–}4\,\mathrm{Hz})$, $N = 42$0, $N = 42$1, $N = 42$2, and $N = 42$3, asymmetry ratios $N = 42$4 and $N = 42$5, and nonlinear features such as Differential Entropy and Sample Entropy. Facial action units are estimated at 2 fps using Py-Feat XGBoost, and voice features include fundamental frequency, shimmer, jitter, harmonicity, formants, and MFCC summaries after Silero-VAD. Whisper large is used to produce Italian text, which is embedded with Sentence-BERT (Italian XXL) into a 768-D vector.

The HRC-SoS description extends this processing logic to online stress-oriented features and classifiers, including R-peak detection by Pan–Tompkins, time-domain HRV features SDNN and RMSSD, frequency-domain HRV via Welch’s method, EDA metrics such as mean skin conductance level and spontaneous SCR count, EEG features such as error-related negativity amplitude and $N = 42$6 bandpower ratio, and pupil dilation relative to baseline (Savur et al., 2019). It also mentions online stress or drowsiness classifiers such as SVM or logistic regression on the feature vector. A plausible implication is that MultiPhysio-HRC was designed not only for retrospective analysis but also for transition toward online physiological computing.

5. Annotations, baseline models, and reported results

Ground truth in MultiPhysio-HRC is questionnaire-based and time-aligned to task blocks (Bussolan et al., 1 Oct 2025). STAI-Y1 provides a 20-item stress measure, NASA-TLX provides six subscales—mental, physical, temporal demand, performance, effort, and frustration—and SAM provides valence, arousal, and dominance on a 1–5 scale. NARS is administered once at the start as a baseline measure of negative attitude toward robots. After each task block, the three self-reports are completed in an instrumented GUI, generating a timestamped entry in metadata.

Baseline supervised learning is reported for both regression and classification (Bussolan et al., 1 Oct 2025). The models are Random Forest, AdaBoost, and XGBoost. Regression targets are normalized STAI-Y1 and NASA-TLX, optimized with

$N = 42$7

whereas classification uses Low/Medium/High labels per subject with cross-entropy loss

$N = 42$8

Validation follows Leave-One-Subject-Out; features are normalized per subject and there is no further feature selection.

The reported regression performance is $N = 42$9 RMSE for STAI-Y1 using physiological features, $27.98 \pm 10.22$0 using EEG, and $27.98 \pm 10.22$1 using voice; for NASA-TLX, the corresponding RMSE values are $27.98 \pm 10.22$2, $27.98 \pm 10.22$3, and $27.98 \pm 10.22$4 (Bussolan et al., 1 Oct 2025). The reported three-way F1 scores are $27.98 \pm 10.22$5 for stress classification using physiological features, $27.98 \pm 10.22$6 using EEG, and $27.98 \pm 10.22$7 using voice; for cognitive load classification they are $27.98 \pm 10.22$8, $27.98 \pm 10.22$9, and $\{\text{Normal, High}\}$0, respectively. Paired Wilcoxon signed-rank tests showed physiological features outperform EEG and voice in regression with $\{\text{Normal, High}\}$1.

The earlier monitoring framework reports complementary results for robot-motion sensitivity rather than cross-task corpus benchmarking (Savur et al., 2019). Heart rate comparison between Fixed and Random trajectories yielded repeated-measures ANOVA $\{\text{Normal, High}\}$2, $\{\text{Normal, High}\}$3, $\{\text{Normal, High}\}$4, with mean HR increasing from $\{\text{Normal, High}\}$5 bpm in Fixed to $\{\text{Normal, High}\}$6 bpm in Random, and a 95% CI for the difference of $\{\text{Normal, High}\}$7. RMSSD differed between low and high acceleration with Wilcoxon signed-rank $\{\text{Normal, High}\}$8, $\{\text{Normal, High}\}$9, moving from $\times \{\text{Fixed, Random}\}$0 ms under low acceleration to $\times \{\text{Fixed, Random}\}$1 ms under high acceleration. Median $\times \{\text{Fixed, Random}\}$2SCR for Random minus Fixed was $\times \{\text{Fixed, Random}\}$3, with 95% CI $\times \{\text{Fixed, Random}\}$4 and $\times \{\text{Fixed, Random}\}$5. Together, these findings indicate that both structured psychometric labeling and event-locked robot stimuli are empirically productive for HRC physiology research.

6. Interpretation, control relevance, and limitations

A recurring empirical point in this research area is that external task performance may remain stable while internal physiological burden changes substantially. A contact-rich pHRI tracing study under 18 combinations of temperature, acoustic noise, and illuminance reported tracing error of approximately $\times \{\text{Fixed, Random}\}$6 mm and completion time of approximately $\times \{\text{Fixed, Random}\}$7 s with no significant dependence on temperature, noise, or illuminance, all $\times \{\text{Fixed, Random}\}$8 (Chen et al., 12 Jun 2026). Physical workload measured by EMG and cognitive workload measured by LHIPA showed no main effects, but autonomic workload indexed by $\times \{\text{Fixed, Random}\}$9 rose from $|\Delta RR| > 300\,\mathrm{ms}$0 at $|\Delta RR| > 300\,\mathrm{ms}$1 to $|\Delta RR| > 300\,\mathrm{ms}$2 at $|\Delta RR| > 300\,\mathrm{ms}$3 and $|\Delta RR| > 300\,\mathrm{ms}$4 at $|\Delta RR| > 300\,\mathrm{ms}$5, with ANOVA $|\Delta RR| > 300\,\mathrm{ms}$6, $|\Delta RR| > 300\,\mathrm{ms}$7, and $|\Delta RR| > 300\,\mathrm{ms}$8. Subjective comfort ratings did not correlate with tracing error or completion time, with $|\Delta RR| > 300\,\mathrm{ms}$9. The interpretation given is compensatory arousal: operators preserved external performance by recruiting additional autonomic resources under thermal stress.

This result is directly relevant to MultiPhysio-HRC because it counters a common misconception that productivity or task accuracy alone is a sufficient proxy for human state. The dataset’s combination of physiological streams, audio-facial cues, and validated self-reports is structured precisely to expose such hidden costs (Bussolan et al., 1 Oct 2025). A plausible implication is that multimodal HRC datasets are most valuable when they enable dissociation between task success and operator burden.

Related work also makes explicit how such measurements can enter the control loop (Chen et al., 12 Jun 2026). The proposed physiology-aware control architecture is Human $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$0 Robot $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$1 Environment sensors $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$2 Signal processing $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$3 Feature extraction $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$4 $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$5 State estimator $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$6 Assistance modulator $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$7 Admittance controller $$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$8 Robot motion. Adaptive planar admittance is expressed as

$$w[n] = 0.54 - 0.46\cos\bigl(\tfrac{2\pi n}{N-1}\bigr), \quad n=0\ldots N-1.$$9

with planar input

$V_2$0

and desired contact force adaptation

$V_2$1

The same framework specifies pipeline latencies of approximately 2 ms for sEMG, 20 ms for EDA, and 10 ms for eye data, aligns features in a 200 ms sliding window, constrains gain adjustments, and uses fallback to nominal gains if data dropouts occur.

The HRC-SoS case study on human comfort versus robot acceleration provides an earlier closed-loop rationale (Savur et al., 2019). Using a UR5e collaborative arm with varied acceleration profiles, BITalino ECG and EDA, accelerometer data, a Pupil Labs headset, and OptiTrack markers, the study collected SDNN, RMSSD, SCL, nSCR, pupil diameter change, subjective comfort ratings, peak acceleration $V_2$2, and average speed. High-acceleration motions with $V_2$3 produced a 25% drop in SDNN and a 40% rise in SCR count relative to low-acceleration trials; pupil dilation increased by approximately $V_2$4 mm under high-jerk profiles, and smoother trajectories were rated as significantly more comfortable with $V_2$5. These mappings fed back into the Compliance subsystem so that a planned high-speed move could be automatically smoothed if the predicted stress index exceeded a comfort threshold.

Open directions stated for MultiPhysio-HRC include adding eye tracking, pupillometry, and motion kinematics; exploring CNN-LSTM and Transformer architectures for multimodal fusion; adapting models to new industrial tasks; and using online incremental learning, few-shot methods, or meta-learning for unseen users (Bussolan et al., 1 Oct 2025). Related design guidelines also emphasize modular ROS or LSL processing nodes, a 3 min neutral-environment baseline for SCL, EMG, and pupil measures, asynchronous fusion with gain updates at 5 Hz, and limiting assistive force changes to less than 10% per second to avoid startle (Chen et al., 12 Jun 2026). Collectively, these directions position MultiPhysio-HRC within a transition from descriptive multimodal monitoring toward personalized, physiology-aware HRC control.