A Near-Raw Talking-Head Video Dataset for Various Computer Vision Tasks

Abstract: Talking-head videos constitute a predominant content type in real-time communication, yet publicly available datasets for video processing research in this domain remain scarce and limited in signal fidelity. In this paper, we open-source a near-raw dataset of 847 talking-head recordings (approximately 212 minutes), each 15\,s in duration, captured from 805 participants using 446 unique consumer webcam devices in their natural environments. All recordings are stored using the FFV1 lossless codec, preserving the camera-native signal -- uncompressed (24.4\%) or MJPEG-encoded (75.6\%) -- without additional lossy processing. Each recording is annotated with a Mean Opinion Score (MOS) and ten perceptual quality tokens that jointly explain 64.4\% of the MOS variance. From this corpus, we curate a stratified benchmarking subset of 120 clips in three content conditions: original, background blur, and background replacement. Codec efficiency evaluation across four datasets and four codecs, namely H.264, H.265, H.266, and AV1, yields VMAF BD-rate savings up to $-71.3\%$ (H.266) relative to H.264, with significant encoder$\times$dataset ($η_p² = .112$) and encoder$\times$content condition ($η_p² = .149$) interactions, demonstrating that both content type and background processing affect compression efficiency. The dataset offers 5$\times$ the scale of the largest prior talking-head webcam dataset (847 vs.\ 160 clips) with lossless signal fidelity, establishing a resource for training and benchmarking video compression and enhancement models in real-time communication.

• The paper introduces 847 clips totaling about 212 minutes from 805 participants and 446 camera models, captured at consumer resolutions and stored with FFV1 to preserve camera-native detail.
• The paper shows that perceptual distortion tokens explain 64.4% of MOS variation, with noise, low resolution, blur, and lighting problems dominating real webcam footage.
• The paper demonstrates that reference fidelity changes codec conclusions: H.266 achieves a 71.3% VMAF BD-rate reduction versus H.264, while lossy VP8 preprocessing reduces measured AV1 and H.266 gains.

Motivation and positioning

Talking-head video dominates real-time communication (RTC) traffic, yet the datasets available for training and benchmarking video processing models on this content are either domain-mismatched or signal-degraded. Standard codec test corpora such as the HEVC common test sequences and UVG contain professionally captured footage that lacks the sensor noise, low-light behavior, and auto-exposure characteristics of consumer webcams. Web-scale face video corpora (VoxCeleb, HDTF, VFHQ) inherit platform compression, and DH-FaceVid-1K standardizes to $512\times512$ face crops, discarding background context. The closest prior resource, VCD, provides 160 clips of 10 s but was captured from the lossy output of webcam firmware (VP8/H.264), embedding double-compression artifacts into the reference signal. The paper addresses this gap with a "near-raw" capture design: camera-native signal, losslessly stored, with no lossy stage beyond camera firmware.

Capture pipeline

The authors built a DirectShow/FFmpeg recording application that opens each UVC webcam at its maximum supported resolution (minimum 1280×720 at 30 fps) and selects the highest-fidelity pixel format: YUYV422 or NV12 when supported, MJPEG as fallback. All frames are encoded with FFV1 Level 3 in Matroska containers, with 16 slices per frame and per-slice CRC-32 checks guaranteeing bit-exact reconstruction. Frame timestamps pass through unmodified; a 256 MB ring buffer absorbs I/O latency. Of the published clips, 24.4% use uncompressed formats and 75.6% use MJPEG — an important caveat, since for those clips the quality ceiling is bounded by intra-camera DCT compression, though the annotation data suggest this is rarely perceptible (blockiness detected in only 1.3% of raw recordings). Camera firmware operations (demosaicing, white balance, gamma, noise reduction) are not bypassable via UVC, so "near-raw" accurately describes the fidelity bound rather than overselling it.

Dataset composition and annotation

From 1,119 recordings collected via Prolific across four motion scenarios (slow body movement, hand counting, text reading, natural call behavior), quality control removed 272 clips, yielding 847 published clips (~212 minutes) from 805 participants using 446 unique camera models — roughly 5× the scale of the largest prior talking-head webcam dataset (847 vs. 160 clips). Resolution distribution is dominated by 720p (60.7%) and 1080p (33.2%), reflecting realistic consumer hardware.

Annotation proceeds along two axes. Overall quality uses ACR-based MOS per ITU-T P.910 with crowdsourced collection (~7 votes/clip, 216 qualified workers). Diagnostic annotations use ten perceptual tokens derived through a three-phase process: free-text distortion description on a 31% sample (2,596 comments), validation on a 200-clip subsample, then full-corpus annotation requiring ≥2 independent assessor selections per token. Dominant distortions are noise (69.8% selection rate), low resolution (61.3%), and lighting/color issues (36.6%). Two results stand out:

• Cross-method reliability: token-paradigm MOS correlates strongly with independently collected ACR MOS ($r = 0.859$ full corpus, $r = 0.893$ on the common subset).
• Diagnostic power: a multiple regression of MOS on token proportions yields $R^2 = 0.644$, with blur and low resolution carrying the largest negative coefficients ($\beta \approx -1.0$).

The near-zero blockiness rate (0.0% in published clips vs. 10.6% in VCD) empirically validates the lossless capture claim. Notably, over 50% of clips have MOS below 3.5, indicating substantial headroom for enhancement research on typical webcam feeds.

Benchmarking subset

A stratified subset of 120 clips (trimmed to 10 s) is organized into three mutually exclusive groups of 40: original talking-head (TH), production-grade background blur (TH-BB), and background replacement (TH-BR). Stratification crosses three MOS bins with four SI×TI quadrants (12 strata), targeting a 25/50/25 low/medium/high MOS split, with a greedy composite score balancing stratum quotas, rare-token coverage, feature-space diversity, participant uniqueness, and structural detail. Token distributions are well balanced across groups, confirming the stratification preserves distortion-type coverage.

Codec efficiency analysis

The authors benchmark H.264 (Intel QSV hardware), H.265 (QSV hardware), VVenC H.266, and libaom AV1 in low-delay fixed-QP configurations, computing BD-rate relative to H.264 under PSNR and VMAF. Key findings:

Dataset	N	H.265	AV1	H.266
HEVC CTC	25	−34.9	−32.6	−65.0
UVG	16	−34.5	−40.0	−70.4
VCD	120	−27.9	−36.3	−63.7
NR-TH	120	−25.9	−42.2	−71.3
NR-TH + VP8	120	−25.6	−35.1	−62.9

(VMAF BD-rate % relative to H.264.) A two-way mixed ANOVA confirms encoder as the dominant factor ($\eta_p^2 = .789$) but reveals a significant encoder×dataset interaction ($\eta_p^2 = .112$): the relative advantage of modern codecs depends on content type. NR-TH yields larger savings for AV1 and H.266 than VCD, while PSNR and VMAF diverge for hardware H.265 — a metric-dependence worth noting for future benchmarking studies.

Background processing matters: the encoder×content-condition interaction is significant ($\eta_p^2 = .149$ under VMAF, $\eta_p^2 = .479$ under PSNR). Background replacement amplifies software-codec gains dramatically — AV1 reaches −58.9% and H.266 −77.1% on TH-BR versus −30.0% and −71.6% on TH — while hardware H.265 benefits far less. Simplified backgrounds are exploited more effectively by modern software encoders.

Source noise selectively penalizes H.265: linear mixed-effects models show no token affects PSNR BD-rate, but under VMAF the noise token significantly degrades H.265's advantage on original content (coefficient +40.4, $p < .001$), attenuated by background processing. AV1 and H.266 show no significant noise sensitivity, maintaining stable ratios to the baseline.

Lossy capture compresses measured gains: pre-encoding NR-TH clips with VP8 at 2500 kbps (simulating WebRTC capture) worsens overall VMAF BD-rate by 5.0 pp ($r = 0.859$0), with significant degradation for AV1 (+7.1 pp) and H.266 (+7.7 pp) but not H.265. The resulting BD-rates closely match VCD's, corroborating that VCD-style references underestimate modern codec efficiency. This is the paper's strongest methodological argument: lossless references are necessary to isolate codec-induced distortion from capture-pipeline artifacts.

Limitations and open questions

Several constraints qualify these results. First, 75.6% of recordings are MJPEG-sourced, so the "lossless" guarantee applies to the storage layer rather than the full signal path; the authors argue camera MJPEG operates at imperceptible quality, supported by the near-zero blockiness detection, but a direct quantification of residual camera-compression impact is not provided. Second, the codec study uses a single hardware vendor's QSV implementations for H.264/H.265 and specific software versions, so conclusions about hardware-versus-software behavior may not generalize across encoder implementations. Third, the VP8 simulation approximates but does not replicate actual WebRTC pipelines (fixed 2500 kbps CBR, realtime preset). Fourth, the dataset's demographic and geographic composition reflects Prolific recruitment, and scenario S04 (natural call behavior) contributes only 72 clips. Finally, the demonstrated analyses cover compression efficiency; the utility claims for super-resolution, restoration, and video quality assessment training remain asserted rather than experimentally validated in this work.

Conclusion

This paper delivers a substantially scaled, fidelity-preserving talking-head webcam corpus with rich subjective annotation and a carefully stratified benchmarking subset. Its empirical contributions are concrete: perceptual tokens explain 64.4% of MOS variance; content type and background processing significantly modulate measured codec efficiency; source-level noise selectively erodes hardware H.265's VMAF-based advantage; and lossy capture demonstrably shrinks the apparent gains of AV1 and H.266. For researchers benchmarking codecs or training enhancement models for RTC, the results make a clear case that reference signal fidelity is not a peripheral concern but a first-order determinant of experimental conclusions.