---
title: 'HARMONIC Dataset: Human–Robot Shared Autonomy'
url: https://www.emergentmind.com/topics/harmonic-dataset
type: topic
---

# HARMONIC Dataset: Human–Robot Shared Autonomy

The HARMONIC Dataset is a comprehensive multimodal corpus focused on human–robot collaboration in shared autonomy scenarios, specifically targeting assistive eating tasks using a 6 degree-of-freedom (DOF) robotic arm. Encompassing synchronized human, robot, and environment data from 24 participants, the dataset has been assembled to enable in-depth study of intent prediction, cognitive state modeling, and the dynamics of shared human–robot control. All primary signals—including eye gaze, egocentric and third-person video, joystick control, electromyography (EMG), and full robot state—are time-aligned and accompanied by derived features (body pose, hand pose, facial landmarks), providing a rich substrate for machine learning and human–robot interaction (HRI) research [1807.11154].

## 1. Experiment Design and Task Protocol

The experimental paradigm is an assistive eating task: each participant is seated before three marshmallows arranged on a plate and operates a Kinova Mico robotic arm via a 2-axis joystick with three discrete mode switches (x–y, z–yaw, pitch–roll). The protocol involves two principal stages per trial: user teleoperation positions the forked arm above a chosen morsel followed by an autonomous fork “spearing” and serving action, triggered by a long-press mode switch.

Crucially, shared autonomy is instantiated as a POMDP framework over a finite goal set $G = \{g_1, g_2, g_3\}$, corresponding to the morsels. Online intent inference maintains a belief $b(g)$, used to blend the human joystick input $u$ and the robot's computed assistive action $a$ via
$$
a_\mathrm{applied} = (1-\gamma) u + \gamma a,
$$
with $\gamma \in [0,1]$ controlling autonomy level (teleoperation: $\gamma=0$, low: $0.33$, high: $0.67$, autonomous: $1.0$). Each of 24 naïve participants performs 5 trials at all 4 assistance levels, yielding 480 total trials and about 5 hours of recorded multimodal data.

## 2. Sensor Modalities and Feature Set

The dataset captures multimodal streams with high temporal precision and spatial accuracy for each trial:

- **Binocular Eye Tracking:** Pupil Labs near-IR dark-pupil system (120 Hz, 640×480 px), with synchronized raw pupil center extraction, per-eye confidence scores, and manual AprilTag-based calibration for egocentric gaze mapping.
- **Egocentric (Scene) Camera:** Pupil Labs RGB at 30 Hz (1280×720 px), with timestamped frame indexing.
- **Third-person Stereo Video:** Stereolabs ZED (left/right, 1920×1080 px, 30 Hz) for whole-body movement capture; no published calibration, but rectification via ZED SDK possible.
- **Joystick Control:** Real-time logging of raw x/y axis inputs, mode state, and assistance information at nominal 120 Hz; full-resampled to a uniform time grid.
- **Surface Electromyography (EMG):** Myo armband, 8 channels (50 Hz), with concurrent IMU (accelerometer, gyroscope, quaternion orientation), present in 21% of trials with >99% coverage when available.
- **Robot State:** Kinova Mico 6-DOF joint positions and velocities (80 Hz); derived forward-kinematics Cartesian positions of all links, with $T_{0\to6}(\theta)$ and $p_\mathrm{tool}$ available at each timestamp.
- **Shared Autonomy Metadata:** Assistance blending coefficients, inferred POMDP goal belief distributions $b(g_i)$, and applied assistive twist $a$.
- **Environmental Ground Truth:** Homogeneous transforms (AprilTag markers) giving the global position of each morsel in the robot frame.

Derived high-level features incorporate:
- 2D human pose (25 OpenPose joints), left/right hand (21 keypoints each), facial landmarks (70 points), computed offline from ZED video streams.

## 3. Data Formats, Organization, and Synchronization

Directory hierarchy is organized per participant and run, with strict subfolder separation:

- `pXXX/` (participant root)
    - `calib/`: camera and pupil calibration CSVs
    - `check/`: inter-block calibration verification
    - `run/run_NNN/`
        - `text_data/`: CSV/YAML for all raw and processed streams (e.g., `gaze_positions.csv`, `ada_joy.csv`, `joint_states.csv`, `robot_position.csv`, EMG, pose, assistance info)
        - `videos/`: all original and processed MP4s, timestamp arrays (`*_timestamps.npy`)
        - `stats/`: YAML with nominal frequency, frame drops, coverage stats
        - `processed/`: overlays and derived feature streams

Synchronization leverages nanosecond-precision timestamps and two video index mappings: `world_index` and `world_index_corrected` for accurate alignment across asynchronous streams. All major time series are aligned to scene video, with code templates for resampling, frame extraction, and overlay.

Example headers (abridged):

| File                        | Key Columns/sample             |
|-----------------------------|-------------------------------|
| gaze_positions.csv           | timestamp, norm_pos_x, norm_pos_y, confidence, world_index, world_index_corrected |
| ada_joy.csv                 | timestamp, mode, joy_x, joy_y |
| myo_emg.csv                 | timestamp, emg0–emg7          |
| joint_states.csv            | timestamp, joint_i_pos, joint_i_vel |
| pose.csv                    | timestamp, joint_1_x, joint_1_y, ..., joint_25_x, joint_25_y |

## 4. Experiment Coverage and Data Quality

- **Population:** 24 participants (13 female, ages 18–45), all non-expert with respect to robotics and teleoperation.
- **Trial Structure:** 4 assistance levels × 5 trials per level × 24 participants = 480 trials.
- **Coverage:** Full video and robot/joystick state in all trials; eye-tracking and EMG have ≤1% frame loss where present. EMG present in approximately 21% of trials due to initialization failures (when present, >99% temporal coverage).
- **Frame Drop Monitoring:** Per-run YAML stats document expected vs. actual data frames and dropped indices; coverage typically ≈95%.

All data signals are intended for high-fidelity multimodal behavioral analysis, with careful timestamping and alignment mechanisms to permit cross-modal integration.

## 5. Example Code and User Guidance

To support downstream analysis, practical code fragments are provided. Typical pre-processing or alignment tasks include:

```python
import pandas as pd, numpy as np, cv2
gaze = pd.read_csv('text_data/gaze_positions.csv')
world_ts = np.load('videos/world_timestamps.npy')
vid = cv2.VideoCapture('videos/world.mp4')
for idx,row in gaze.iterrows():
    f = int(row['world_index_corrected'])
    vid.set(cv2.CAP_PROP_POS_FRAMES, f)
    ret,frame = vid.read()
    x_px = int(row['norm_pos_x'] * frame.shape[1])
    y_px = int(row['norm_pos_y'] * frame.shape[0])
    cv2.circle(frame,(x_px,y_px),5,(0,0,255),-1)
```
For resampling:
```python
t0 = world_ts[0]
t_common = t0 + np.arange(0,int((world_ts[-1]-t0)/1e9*30))*1e9/30
emg = pd.read_csv('text_data/myo_emg.csv')
emg['timestamp'] = emg['timestamp'].astype(np.int64)
emg.set_index('timestamp',inplace=True)
emg_rs = emg.reindex(t_common, method='nearest')
```

Best practices include strict usage of original nanosecond timestamps for all cross-modal synchronizations and forward-filling control/EMG modalities to mask gaps.

## 6. Research Applications and Baseline Results

The HARMONIC Dataset is designed for several core research uses:

- **Intention Prediction:** Fusion of gaze and EMG predicts goal probabilities in shared autonomy POMDP belief update frameworks.
- **Human Policy Modeling:** Data-driven characterization of human adaptation under different autonomy levels.
- **Cognitive State Assessment:** Pupil dynamics in teleoperation are leveraged for cognitive load estimation.
- **Learning Eye–Hand–Control Couplings:** Modeling and imitation learning of tightly coupled eye, hand, and control device signals for assistive robotics.

Prior analysis using subsets of HARMONIC demonstrated intention inference accuracy ≈85% (three-way goal prediction) and F1 ≈0.75 for manipulation error detection from gaze signals.

## 7. Access, Licensing, and Community Use

Multiple dataset subsets are provided for ease of access:

- Full dataset: ~68 GB (`harmonic_data.tar.gz`)
- Minimal (CSV+video+stats): ~15 GB (`harmonic_minimal.tar.gz`)
- Text only: ~4 GB (`harmonic_text.tar.gz`)
- Single-participant sample: ~300 MB

The data is publicly available at http://harp.ri.cmu.edu/harmonic. No specific licensing information is present in the primary publication, but data is supplied in standard, human-readable formats for broad reuse.

---

The HARMONIC Dataset constitutes an unparalleled resource for empirical HRI research, enabling fine-grained analysis and modeling of shared autonomy, intent inference, and behavioral coordination in the context of assistive robotics [1807.11154].

Source: https://www.emergentmind.com/topics/harmonic-dataset