SenseSeek: Multimodal Info-Seeking Dataset

• SenseSeek is a comprehensive multimodal dataset that captures defined search stages using consumer-grade sensors like EEG, EDA, PUPIL, and MOTION.
• The dataset enables analysis of cognitive intent by segmenting search tasks into stages such as query formulation, submission, and relevance judgment.
• It provides actionable insights into multimodal fusion, revealing stage-specific physiological patterns that aid in evaluating search behavior.

Searching arXiv for relevant papers on SenseSeek and closely related research areas. SenseSeek is a public, research-oriented dataset for studying information seeking and web search through multimodal sensing. It combines controlled search tasks that are decomposed into fine-grained search stages with simultaneous recordings from multiple consumer-grade sensors; the released corpus contains data from 20 participants, 235 stimulated search trials, and 940 phases spanning Realization of Information Need (IN), Query Formulation (QF), Query Submission by Typing (QS-T) or Speaking (QS-S), and Relevance Judgment by Reading (RJ-R) or Listening (RJ-L), together with Electrodermal Activities (EDA), Electroencephalogram (EEG), PUPIL, GAZE, and MOTION measurements and 258 extracted features (Ji et al., 20 Jul 2025). To the authors’ knowledge, it is the first dataset that characterizes the multiple stages involved in information seeking with physiological signals collected from multiple sensors (Ji et al., 20 Jul 2025). In a broader interpretive sense, the name also usefully evokes a class of systems in which sensing, belief updating, and action selection are tightly coupled across human–computer interaction, multimedia retrieval, semantic navigation, and multi-robot source seeking; the active sensing literature provides a general formal backdrop for that synthesis (Varotto et al., 2021).

1. Experimental scope and search-stage decomposition

SenseSeek was created to address three gaps at once: the cognitive complexity of information seeking, the increasing availability of passive consumer-grade sensing, and the absence of reusable multimodal datasets organized around explicit, labeled search stages rather than generic media-consumption or emotion tasks (Ji et al., 20 Jul 2025). The study used 12 mock search tasks per participant, drawn from TREC 2002–2004 topic sets and refined via the TREC InformationNeed collection. All tasks were understanding-type information needs, paired with one predefined highly relevant result presented either as text or as audio via text-to-speech, and designed to reduce strong affective confounds by excluding controversial topics such as war, crisis, or politics (Ji et al., 20 Jul 2025).

Each trial followed a fixed sequence: a 4 s blank screen with fixation cross, topic display and self-ratings, backstory presentation for IN, a 10 s thinking period for QF, query submission either by typing or speaking, presentation of a result for RJ-R or RJ-L, and a binary comprehension or judgment question with additional self-ratings (Ji et al., 20 Jul 2025). Between QF, QS, and RJ, the protocol inserted a 4-second gap with fixation to reduce physiological carry-over between stages (Ji et al., 20 Jul 2025). Five of 240 trials were removed for errors, leaving 235 usable trials (Ji et al., 20 Jul 2025).

Stage	Meaning
EYEOPEN	Baseline with eyes open
EYECLOSE	Baseline with eyes closed
IN	Realization of Information Need
QF	Query Formulation
QS-T / QS-S	Query Submission by Typing or Speaking
RJ-R / RJ-L	Relevance Judgment by Reading or Listening

The participant pool comprised 20 individuals, including 12 male and 8 female participants, with most between 25 and 34 years old and with at least professional working proficiency in English (Ji et al., 20 Jul 2025). This design yielded 940 search-stage segments for EEG, head motion, wrist motion, and EDA, plus 40 baseline segments; eye tracking failed for two participants, leaving 859 stage instances for PUPIL and GAZE (Ji et al., 20 Jul 2025).

2. Instrumentation, preprocessing, and feature space

The sensing stack combined an Empatica E4 wristband, an Emotiv EPOC X EEG headset with integrated IMU, and a Tobii Fusion screen-mounted eye tracker (Ji et al., 20 Jul 2025). EDA was sampled at 4 Hz, wrist motion at 32 Hz, EEG at 128 Hz across 14 channels, and gaze and pupil at 60 Hz (Ji et al., 20 Jul 2025). Qualtrics recorded onset and offset timestamps for all stages with JavaScript, after which timestamps were converted to ISO 8601 with millisecond precision and used to segment each continuous sensor stream (Ji et al., 20 Jul 2025).

Modality	Device	Sampling
EDA	Empatica E4	4 Hz
Wrist MOTION	Empatica E4	32 Hz
EEG	Emotiv EPOC X	128 Hz
Head MOTION	Emotiv EPOC X	128 Hz
GAZE / PUPIL	Tobii Fusion	60 Hz

The preprocessing pipeline was modality-specific. EEG was re-referenced to common average, band-pass filtered from 1 to 40 Hz, cleaned with Autoreject and ICA/ICLabel, and converted into Welch PSD features over $\Theta$ (4–8 Hz), $\Alpha$ (8–13 Hz), $\Beta$ (13–30 Hz), and $LowGamma$ (25–40 Hz) bands (Ji et al., 20 Jul 2025). EDA was median-filtered, standardized with z-score normalization, and decomposed by cvxEDA into tonic Skin Conductance Level and phasic Skin Conductance Response components (Ji et al., 20 Jul 2025). Wrist motion was reduced to acceleration magnitude $|a|=\sqrt{x^2+y^2+z^2}$ and smoothed with a rolling median filter; pupil signals were blink-cleaned, interpolated, low-pass filtered, and converted into Relative Pupil Dilation using participant-specific EYEOPEN baselines (Ji et al., 20 Jul 2025).

The released feature matrix contains 258 features. These are partitioned into 31 EDA features, 180 EEG features, 25 PUPIL features, 13 MOTION features, and 6 GAZE features (Ji et al., 20 Jul 2025). Physiological features were baseline-normalized per participant, whereas behavioral features were standardized across each participant’s task distribution (Ji et al., 20 Jul 2025). The dataset also includes gaze-annotated screen recordings, topic texts, backstories, search results in text and audio, self-reports, and task responses (Ji et al., 20 Jul 2025).

3. Cognitive-stage effects and baseline discrimination performance

Baseline analyses were organized around three questions: the impact of cognitive intent across stages, the effect of interaction modality, and the detectability of search stages from multimodal data (Ji et al., 20 Jul 2025). EEG showed the clearest stage-level differences in the alpha band, with significant effects at many electrodes and peak Friedman statistics reported at P7, O1, and F4 (Ji et al., 20 Jul 2025). Beta-band power also varied across stages, whereas theta and low-gamma effects were weaker and more spatially selective (Ji et al., 20 Jul 2025). The interpretation advanced in the study is that alpha and beta shifts reflect changes in attention allocation, visual processing, and cognitive effort across information-need realization, formulation, submission, and relevance judgment (Ji et al., 20 Jul 2025).

Pupil dilation was especially sensitive to query submission. Mean Relative Pupil Dilation differed significantly across stages, and both QS-S and QS-T produced substantially higher RPD than EYEOPEN, IN, QF, RJ-R, and RJ-L in multiple post-hoc comparisons (Ji et al., 20 Jul 2025). By contrast, EDA yielded a more selective result: mixed EDA mean and SCL mean were non-significant across stages, whereas SCR mean was significant, indicating that phasic arousal carried more stage-discriminative signal than tonic arousal in this protocol (Ji et al., 20 Jul 2025).

The comparison between EYEOPEN, IN, and RJ-R is particularly important because all three involve screen viewing. Here the study found EEG differences despite superficial modality similarity, and pupil time courses separated IN from RJ-R even when average RPD did not separate EYEOPEN from either condition (Ji et al., 20 Jul 2025). This supports the dataset’s central claim that cognitive intent, not only overt input or output modality, modulates physiological response during search (Ji et al., 20 Jul 2025).

Stage classification used leave-one-participant-out evaluation with six classes: IN, QF, QS-T, QS-S, RJ-R, and RJ-L (Ji et al., 20 Jul 2025). PUPIL was the strongest single modality, reaching accuracy 0.585 $(\pm 0.151)$ and F1-macro 0.574 $(\pm 0.152)$ with 4 s windows and 2 s overlap; EEG was more modest at 0.328 accuracy and 0.252 F1-macro, while EDA was near baseline (Ji et al., 20 Jul 2025). The best decision-fusion configuration excluded EDA and combined EEG, PUPIL, MOTION, and GAZE, reaching accuracy 0.671 $(\pm 0.147)$ and F1-macro 0.645 $(\pm 0.161)$ (Ji et al., 20 Jul 2025). A plausible implication is that SenseSeek is at least as much a benchmark for multimodal fusion design as it is a dataset for individual signal analysis.

4. Query language, search behavior, and the semantics of seeking

SenseSeek studies search physiologically, but adjacent work on sound search examines the textual form of user queries and thereby clarifies what a sensing-aware search system must interpret. In a survey of 94 participants covering 706 tasks, users formulated queries for an unrestricted hypothetical sound search engine; in parallel, Freesound query logs from April to June 2024 provided approximately 9 million real-world search requests (Weck et al., 2024). The survey queries were generally longer than Freesound queries, and both datasets were dominated by keyword-based formulations rather than full sentences (Weck et al., 2024).

The factors most strongly influencing the survey queries were the primary sound source, intended usage, perceived location, and the number of sound sources (Weck et al., 2024). Real-world sound search also revealed specialized vocabularies, including utterances and vocables (“oh no”, “yeah”, “hmm”), production jargon (“riser”, “one shot”, “stab”, “stinger”, “bumper”), abbreviations such as “atmo” and “bgm”, and intended-use queries such as “error”, “success”, “correct answer”, “alert”, “button click”, “game over”, and “jumpscare” (Weck et al., 2024). These findings matter for SenseSeek because they show that realistic search behavior is typically terse, compositional, and domain-coded rather than caption-like.

A plausible implication is that SenseSeek-style search interfaces should not assume sentence-form natural language as the dominant retrieval input. They should instead support short keyword strings, aspect-based refinement, and multi-source compositions, because longer queries appear in unconstrained settings only when users believe the system can interpret them effectively (Weck et al., 2024). This complements the dataset’s stage annotations: the physiological side characterizes how people search, while the query-language side characterizes what they actually say or type during search.

5. Semantic priors, latent retrieval, and probabilistic seeking

The broader “sense-and-seek” interpretation becomes clearer in semantic and latent-space search systems. SEEK, introduced for object-goal navigation in inspection tasks, combines a Dynamic Scene Graph with a Relational Semantic Network that estimates the probability of finding a target object across spatial elements, then plans over rooms with a probabilistic MDP solved by value iteration (Ginting et al., 2024). In Habitat/Matterport3D experiments, SEEK reported SPL 0.96 $(0.12)$ on fixed objects and 0.84 $\Alpha$0 on movable objects, outperforming the GPT-4 Planner baseline at 0.84 $\Alpha$1 and 0.81 $\Alpha$2 respectively (Ginting et al., 2024). In a real-world office deployment on a Spot robot, it guided the robot to the nearest fire extinguisher with SPL $\Alpha$3 while running onboard in real time (Ginting et al., 2024).

A distinct but related archive-scale formulation appears in reverse radio spectrogram search. There, a $\Alpha$4-Variational Autoencoder is trained on 16 $\Alpha$5 256 radio spectrogram snippets, yielding a latent dimension of 5; a positional embedding layer adds frequency information through a 4-dimensional embedding, and cosine similarity is used to retrieve spectrogram patches with similar morphology (Ma et al., 2023). On synthetic clustering tests, the $\Alpha$6-VAE achieved silhouette score $\Alpha$7, cluster metric $\Alpha$8, and disentanglement score $\Alpha$9, outperforming naive flattening, SIFT/BoW, ResNet-50, and a plain autoencoder (Ma et al., 2023).

Taken together, these systems suggest a generalized SenseSeek pattern: maintain a structured latent or semantic representation, update it with evidence, and use it for non-myopic search over a large action or archive space. That interpretation is broader than the dataset’s literal scope, but it is consistent with the neighboring literature’s recurring combination of sensing, representation learning, and guided seeking (Ginting et al., 2024).

6. Active sensing and source-seeking lineage

Within robotics, the most direct conceptual analogue is Active Position Estimation, defined as the task of localizing one or more targets using one or more sensing platforms under uncertainty, with control policies that are either information-seeking, task-driven, or hybrid (Varotto et al., 2021). The survey formalizes target dynamics, platform dynamics, detection probability, Bayes filtering, and objective functions such as A-optimality, D-optimality, mutual information, KL divergence, entropy minimization, probability-of-detection, minimum-time search, and distance-to-estimate criteria (Varotto et al., 2021). A plausible implication is that “SenseSeek” can be read as an application-facing label for this broader active sensing stack.

One instantiation is distributed information-based source seeking. A team of mobile sensors estimates an unknown source position $\Beta$0 from local range-based measurements $\Beta$1, updates that estimate with an EKF, and moves to maximize Fisher information about the source (Zhang et al., 2022). The main loss is the A-optimality criterion

$\Beta$2

and the paper also provides a fully distributed version using consensus-based EKF and consensus-based FIM estimation over a sparse communication graph (Zhang et al., 2022). Simulation and physical Turtlebot-like experiments with light sensors demonstrate the characteristic behavior of first spreading to improve geometry and then converging smoothly toward the source (Zhang et al., 2022).

A second instantiation is unicycle source seeking with 3D-printed flexible graphene-based piezoresistive airflow sensors. There the control law projects the gradient of a scalar field onto nonholonomic unicycle dynamics,

$\Beta$3

and the authors prove asymptotic convergence to the source for strictly concave fields; under partial sensor failure, they replace direct gradient sensing with extremum-seeking control and still prove convergence of averaged trajectories in the quadratic case (Li et al., 2021). The same sense-and-seek coupling appears here: sensing defines the field estimate, and motion is directly driven by that estimate.

A third instantiation appears in multi-agent acoustic localization, where robots can only sense while stationary and therefore operate as a stochastic hybrid system alternating between measurement mode and movement mode (Sorge et al., 16 Oct 2025). The system uses recursive Bayesian estimation for step length and direction of arrival, supports both single-source bearing-rigid formation control and multi-source independent search, and shares explored areas among agents in the multi-source case (Sorge et al., 16 Oct 2025). The listening-then-moving architecture is especially notable because it makes sensor corruption by self-motion part of the control design rather than an afterthought (Sorge et al., 16 Oct 2025).

Across these robotic examples, the recurrent structure is the same: represent uncertainty explicitly, extract informative observations, and make motion or action choices that improve both state estimation and task completion. This suggests that the dataset-level SenseSeek and the robotics-level sense-and-seek lineage are linked by a common systems idea: search is not separable from sensing, because sensing determines what search policy is rational, and search in turn reshapes what can be sensed (Varotto et al., 2021).