---
title: 'SenseSeek: Multimodal Info-Seeking Dataset'
url: https://www.emergentmind.com/topics/senseseek
type: topic
---

# SenseSeek: Multimodal Info-Seeking Dataset

Searching arXiv for relevant papers on SenseSeek and closely related research areas.
SenseSeek is a public, research-oriented dataset for studying information seeking and web search through multimodal sensing. It combines controlled search tasks that are decomposed into fine-grained search stages with simultaneous recordings from multiple consumer-grade sensors; the released corpus contains data from 20 participants, 235 stimulated search trials, and 940 phases spanning Realization of Information Need (IN), Query Formulation (QF), Query Submission by Typing (QS-T) or Speaking (QS-S), and Relevance Judgment by Reading (RJ-R) or Listening (RJ-L), together with Electrodermal Activities (EDA), Electroencephalogram (EEG), PUPIL, GAZE, and MOTION measurements and 258 extracted features [2507.14792]. To the authors’ knowledge, it is the first dataset that characterizes the multiple stages involved in information seeking with physiological signals collected from multiple sensors [2507.14792]. In a broader interpretive sense, the name also usefully evokes a class of systems in which sensing, belief updating, and action selection are tightly coupled across human–computer interaction, multimedia retrieval, semantic navigation, and multi-robot source seeking; the active sensing literature provides a general formal backdrop for that synthesis [2112.02381].

## 1. Experimental scope and search-stage decomposition

SenseSeek was created to address three gaps at once: the cognitive complexity of information seeking, the increasing availability of passive consumer-grade sensing, and the absence of reusable multimodal datasets organized around explicit, labeled search stages rather than generic media-consumption or emotion tasks [2507.14792]. The study used 12 mock search tasks per participant, drawn from TREC 2002–2004 topic sets and refined via the TREC InformationNeed collection. All tasks were understanding-type information needs, paired with one predefined highly relevant result presented either as text or as audio via text-to-speech, and designed to reduce strong affective confounds by excluding controversial topics such as war, crisis, or politics [2507.14792].

Each trial followed a fixed sequence: a 4 s blank screen with fixation cross, topic display and self-ratings, backstory presentation for IN, a 10 s thinking period for QF, query submission either by typing or speaking, presentation of a result for RJ-R or RJ-L, and a binary comprehension or judgment question with additional self-ratings [2507.14792]. Between QF, QS, and RJ, the protocol inserted a 4-second gap with fixation to reduce physiological carry-over between stages [2507.14792]. Five of 240 trials were removed for errors, leaving 235 usable trials [2507.14792].

| Stage | Meaning |
|---|---|
| EYEOPEN | Baseline with eyes open |
| EYECLOSE | Baseline with eyes closed |
| IN | Realization of Information Need |
| QF | Query Formulation |
| QS-T / QS-S | Query Submission by Typing or Speaking |
| RJ-R / RJ-L | Relevance Judgment by Reading or Listening |

The participant pool comprised 20 individuals, including 12 male and 8 female participants, with most between 25 and 34 years old and with at least professional working proficiency in English [2507.14792]. This design yielded 940 search-stage segments for EEG, head motion, wrist motion, and EDA, plus 40 baseline segments; eye tracking failed for two participants, leaving 859 stage instances for PUPIL and GAZE [2507.14792].

## 2. Instrumentation, preprocessing, and feature space

The sensing stack combined an Empatica E4 wristband, an Emotiv EPOC X EEG headset with integrated IMU, and a Tobii Fusion screen-mounted eye tracker [2507.14792]. EDA was sampled at 4 Hz, wrist motion at 32 Hz, EEG at 128 Hz across 14 channels, and gaze and pupil at 60 Hz [2507.14792]. Qualtrics recorded onset and offset timestamps for all stages with JavaScript, after which timestamps were converted to ISO 8601 with millisecond precision and used to segment each continuous sensor stream [2507.14792].

| Modality | Device | Sampling |
|---|---|---|
| EDA | Empatica E4 | 4 Hz |
| Wrist MOTION | Empatica E4 | 32 Hz |
| EEG | Emotiv EPOC X | 128 Hz |
| Head MOTION | Emotiv EPOC X | 128 Hz |
| GAZE / PUPIL | Tobii Fusion | 60 Hz |

The preprocessing pipeline was modality-specific. EEG was re-referenced to common average, band-pass filtered from 1 to 40 Hz, cleaned with Autoreject and ICA/ICLabel, and converted into Welch PSD features over $\Theta$ (4–8 Hz), $\Alpha$ (8–13 Hz), $\Beta$ (13–30 Hz), and $LowGamma$ (25–40 Hz) bands [2507.14792]. EDA was median-filtered, standardized with z-score normalization, and decomposed by cvxEDA into tonic Skin Conductance Level and phasic Skin Conductance Response components [2507.14792]. Wrist motion was reduced to acceleration magnitude $|a|=\sqrt{x^2+y^2+z^2}$ and smoothed with a rolling median filter; pupil signals were blink-cleaned, interpolated, low-pass filtered, and converted into Relative Pupil Dilation using participant-specific EYEOPEN baselines [2507.14792].

The released feature matrix contains 258 features. These are partitioned into 31 EDA features, 180 EEG features, 25 PUPIL features, 13 MOTION features, and 6 GAZE features [2507.14792]. Physiological features were baseline-normalized per participant, whereas behavioral features were standardized across each participant’s task distribution [2507.14792]. The dataset also includes gaze-annotated screen recordings, topic texts, backstories, search results in text and audio, self-reports, and task responses [2507.14792].

## 3. Cognitive-stage effects and baseline discrimination performance

Baseline analyses were organized around three questions: the impact of cognitive intent across stages, the effect of interaction modality, and the detectability of search stages from multimodal data [2507.14792]. EEG showed the clearest stage-level differences in the alpha band, with significant effects at many electrodes and peak Friedman statistics reported at P7, O1, and F4 [2507.14792]. Beta-band power also varied across stages, whereas theta and low-gamma effects were weaker and more spatially selective [2507.14792]. The interpretation advanced in the study is that alpha and beta shifts reflect changes in attention allocation, visual processing, and cognitive effort across information-need realization, formulation, submission, and relevance judgment [2507.14792].

Pupil dilation was especially sensitive to query submission. Mean Relative Pupil Dilation differed significantly across stages, and both QS-S and QS-T produced substantially higher RPD than EYEOPEN, IN, QF, RJ-R, and RJ-L in multiple post-hoc comparisons [2507.14792]. By contrast, EDA yielded a more selective result: mixed EDA mean and SCL mean were non-significant across stages, whereas SCR mean was significant, indicating that phasic arousal carried more stage-discriminative signal than tonic arousal in this protocol [2507.14792].

The comparison between EYEOPEN, IN, and RJ-R is particularly important because all three involve screen viewing. Here the study found EEG differences despite superficial modality similarity, and pupil time courses separated IN from RJ-R even when average RPD did not separate EYEOPEN from either condition [2507.14792]. This supports the dataset’s central claim that cognitive intent, not only overt input or output modality, modulates physiological response during search [2507.14792].

Stage classification used leave-one-participant-out evaluation with six classes: IN, QF, QS-T, QS-S, RJ-R, and RJ-L [2507.14792]. PUPIL was the strongest single modality, reaching accuracy 0.585 $(\pm 0.151)$ and F1-macro 0.574 $(\pm 0.152)$ with 4 s windows and 2 s overlap; EEG was more modest at 0.328 accuracy and 0.252 F1-macro, while EDA was near baseline [2507.14792]. The best decision-fusion configuration excluded EDA and combined EEG, PUPIL, MOTION, and GAZE, reaching accuracy 0.671 $(\pm 0.147)$ and F1-macro 0.645 $(\pm 0.161)$ [2507.14792]. A plausible implication is that SenseSeek is at least as much a benchmark for multimodal fusion design as it is a dataset for individual signal analysis.

## 4. Query language, search behavior, and the semantics of seeking

SenseSeek studies search physiologically, but adjacent work on sound search examines the textual form of user queries and thereby clarifies what a sensing-aware search system must interpret. In a survey of 94 participants covering 706 tasks, users formulated queries for an unrestricted hypothetical sound search engine; in parallel, Freesound query logs from April to June 2024 provided approximately 9 million real-world search requests [2410.08324]. The survey queries were generally longer than Freesound queries, and both datasets were dominated by keyword-based formulations rather than full sentences [2410.08324].

The factors most strongly influencing the survey queries were the primary sound source, intended usage, perceived location, and the number of sound sources [2410.08324]. Real-world sound search also revealed specialized vocabularies, including utterances and vocables (“oh no”, “yeah”, “hmm”), production jargon (“riser”, “one shot”, “stab”, “stinger”, “bumper”), abbreviations such as “atmo” and “bgm”, and intended-use queries such as “error”, “success”, “correct answer”, “alert”, “button click”, “game over”, and “jumpscare” [2410.08324]. These findings matter for SenseSeek because they show that realistic search behavior is typically terse, compositional, and domain-coded rather than caption-like.

A plausible implication is that SenseSeek-style search interfaces should not assume sentence-form natural language as the dominant retrieval input. They should instead support short keyword strings, aspect-based refinement, and multi-source compositions, because longer queries appear in unconstrained settings only when users believe the system can interpret them effectively [2410.08324]. This complements the dataset’s stage annotations: the physiological side characterizes how people search, while the query-language side characterizes what they actually say or type during search.

## 5. Semantic priors, latent retrieval, and probabilistic seeking

The broader “sense-and-seek” interpretation becomes clearer in semantic and latent-space search systems. SEEK, introduced for object-goal navigation in inspection tasks, combines a Dynamic Scene Graph with a Relational Semantic Network that estimates the probability of finding a target object across spatial elements, then plans over rooms with a probabilistic MDP solved by value iteration [2405.09822]. In Habitat/Matterport3D experiments, SEEK reported SPL 0.96 $(0.12)$ on fixed objects and 0.84 $(0.23)$ on movable objects, outperforming the GPT-4 Planner baseline at 0.84 $(0.26)$ and 0.81 $(0.23)$ respectively [2405.09822]. In a real-world office deployment on a Spot robot, it guided the robot to the nearest fire extinguisher with SPL $\approx 0.93$ while running onboard in real time [2405.09822].

A distinct but related archive-scale formulation appears in reverse radio spectrogram search. There, a $\beta$-Variational Autoencoder is trained on 16 $\times$ 256 radio spectrogram snippets, yielding a latent dimension of 5; a positional embedding layer adds frequency information through a 4-dimensional embedding, and cosine similarity is used to retrieve spectrogram patches with similar morphology [2302.13854]. On synthetic clustering tests, the $\beta$-VAE achieved silhouette score $-0.12 \pm 0.01$, cluster metric $2.16 \pm 0.06$, and disentanglement score $0.88 \pm 0.04$, outperforming naive flattening, SIFT/BoW, ResNet-50, and a plain autoencoder [2302.13854].

Taken together, these systems suggest a generalized SenseSeek pattern: maintain a structured latent or semantic representation, update it with evidence, and use it for non-myopic search over a large action or archive space. That interpretation is broader than the dataset’s literal scope, but it is consistent with the neighboring literature’s recurring combination of sensing, representation learning, and guided seeking [2405.09822].

## 6. Active sensing and source-seeking lineage

Within robotics, the most direct conceptual analogue is Active Position Estimation, defined as the task of localizing one or more targets using one or more sensing platforms under uncertainty, with control policies that are either information-seeking, task-driven, or hybrid [2112.02381]. The survey formalizes target dynamics, platform dynamics, detection probability, Bayes filtering, and objective functions such as A-optimality, D-optimality, mutual information, KL divergence, entropy minimization, probability-of-detection, minimum-time search, and distance-to-estimate criteria [2112.02381]. A plausible implication is that “SenseSeek” can be read as an application-facing label for this broader active sensing stack.

One instantiation is distributed information-based source seeking. A team of mobile sensors estimates an unknown source position $q \in \mathbb{R}^k$ from local range-based measurements $y_i=h_i(p_i,q)+\nu_i$, updates that estimate with an EKF, and moves to maximize Fisher information about the source [2209.09421]. The main loss is the A-optimality criterion
$$
L(\mathbf{p},q)=\mathrm{Tr}\left[\left(\nabla_q H(\mathbf{p},q)\,\nabla_q H(\mathbf{p},q)^\top\right)^{-1}\right],
$$
and the paper also provides a fully distributed version using consensus-based EKF and consensus-based FIM estimation over a sparse communication graph [2209.09421]. Simulation and physical Turtlebot-like experiments with light sensors demonstrate the characteristic behavior of first spreading to improve geometry and then converging smoothly toward the source [2209.09421].

A second instantiation is unicycle source seeking with 3D-printed flexible graphene-based piezoresistive airflow sensors. There the control law projects the gradient of a scalar field onto nonholonomic unicycle dynamics,
$$
u=k_1\langle \vec{v}(\theta),\nabla J\rangle,\qquad
\omega=-k_2\langle \vec{v}(\theta),\nabla J^\perp\rangle,
$$
and the authors prove asymptotic convergence to the source for strictly concave fields; under partial sensor failure, they replace direct gradient sensing with extremum-seeking control and still prove convergence of averaged trajectories in the quadratic case [2104.14267]. The same sense-and-seek coupling appears here: sensing defines the field estimate, and motion is directly driven by that estimate.

A third instantiation appears in multi-agent acoustic localization, where robots can only sense while stationary and therefore operate as a stochastic hybrid system alternating between measurement mode and movement mode [2510.14849]. The system uses recursive Bayesian estimation for step length and direction of arrival, supports both single-source bearing-rigid formation control and multi-source independent search, and shares explored areas among agents in the multi-source case [2510.14849]. The listening-then-moving architecture is especially notable because it makes sensor corruption by self-motion part of the control design rather than an afterthought [2510.14849].

Across these robotic examples, the recurrent structure is the same: represent uncertainty explicitly, extract informative observations, and make motion or action choices that improve both state estimation and task completion. This suggests that the dataset-level SenseSeek and the robotics-level sense-and-seek lineage are linked by a common systems idea: search is not separable from sensing, because sensing determines what search policy is rational, and search in turn reshapes what can be sensed [2112.02381].

Source: https://www.emergentmind.com/topics/senseseek