MouseTracks: Multi-Domain Tracking Systems
- MouseTracks is a family of trajectory-centric systems that reconstruct and analyze movement in both animal studies and human-computer interfaces using multi-view video and cursor data.
- These systems employ advanced techniques like deep learning, 3D pose reconstruction, non-linear optimization, and neural models to extract precise behavioral metrics and segmentation outcomes.
- Applications span automated multi-animal tracking in laboratory settings, cursor-based saliency estimation in HCI, and behavioral biometrics for continuous authentication and bot detection in security.
Searching arXiv for recent and relevant papers on “MouseTracks” and closely related mouse-tracking usages.
MouseTracks is a term used in several research literatures for systems that reconstruct, track, or analyze trajectories associated either with laboratory mice in video or with human-operated computer mice in graphical interfaces. In laboratory-animal vision, it denotes pipelines for continuous, 3D body and body-part trajectories, multi-object home-cage tracking, and space-time instance segmentation; in HCI, computer vision, and security, it denotes capture and modeling of cursor trajectories for saliency estimation, behavioral biometrics, profiling, bot detection, and continuous authentication (Hellwich et al., 19 Jan 2025, Lyudvichenko et al., 2019, Khan et al., 2022).
1. Scope and principal meanings
The term is not restricted to a single technical object. In the animal-behavior literature, MouseTracks names systems for tracking the pose, identity, or segmentation masks of one or more mice across time. In the HCI and security literature, it names systems built from mouse dynamics, that is, time-stamped cursor coordinates, clicks, and related events, typically treated as a behavioral signal rather than as direct content (Hellwich et al., 19 Jan 2025, Oberhauser et al., 10 Jul 2025, Khan et al., 2022).
| Research domain | Typical input | Typical output |
|---|---|---|
| Laboratory-mouse tracking | Multi-view video, body-part detections, bounding boxes, masks, or events | 3D trajectories, instance masks, tracklets, identities |
| Visual attention and saliency | Mouse-contingent video viewing and cursor positions | Per-frame saliency maps |
| Security and behavioral biometrics | Browser or desktop mouse events, clicks, timestamps | User recognition, validation, demographic inference, bot scores |
This distribution of meanings suggests that MouseTracks is best understood as a family of trajectory-centric systems rather than a single standardized framework. A common denominator is the use of temporally ordered motion observations to recover latent structure: animal pose, object identity, human attention, user identity, or behavioral class.
2. Three-dimensional mouse pose and body-part reconstruction
In animal behavior analysis, MouseTracks has been used to describe a full pipeline for reconstructing continuous, 3D body and body-part trajectories of a mouse from incompletely observed multi-view video, with the explicit goal of enabling more precise behavior analysis such as how and when the mouse manipulates lock-box mechanisms (Hellwich et al., 19 Jan 2025). The pipeline assumes at least three cameras, uses global exterior camera orientation derived from known lock-box coordinates, obtains 2D body-part detections from a 2D pose estimator such as DeepLabCut, and estimates 3D positions through triangulation and bundle adjustment. Its core distinction is that it does not merely triangulate frame by frame: it combines a rigid 3D mouse model, deep-learned body-part movements, and a global motion-track smoothness constraint in a non-linear least-squares objective solved with Ceres. The rigid template specifies biologically reasonable canonical coordinates for the nose tip, ears, paws, and tail root; the pose at time is encoded as a 6-DOF state vector consisting of a Rodrigues rotation vector and translation. Smoothness is imposed with cubic spline interpolation across neighboring times, and the deep component is proposed as a transformer with experiments on simulated data suggesting that LSTM networks may work better in practice. The paper explicitly states that it is a conceptual work and does not yet report detailed quantitative results.
A closely related but monocular line of work infers 3D mouse pose, including the limbs and feet, from single top-down video (Hu et al., 2021). That system uses an SSD detector, a Stacked Hourglass CNN for 2D keypoints, and a kinematic skeleton with fixed bone lengths, joint-angle constraints, and a 5-component Gaussian Mixture Model pose prior. The mouse is constrained to lie at a fixed distance from the camera, thereby removing the scale-depth ambiguity. On a multiview ground-truth dataset, the inferred poses are reported to be accurate enough to estimate stride length even when the feet are mostly occluded, with average error below $10$ mm for each joint in 3D. On gait data, stride lengths derived from the monocular 3D pose were statistically indistinguishable from DigiGait under a two-way ANOVA, with and . The same study reports that 3D joint angles yielded classification accuracy for age, background, and knockout status on 16 held-out animals, indicating that full 3D pose can function as a compact behavioral representation rather than merely a geometric reconstruction (Hu et al., 2021).
3. Multi-animal tracking, segmentation, and identity maintenance
For interacting animals, MouseTracks expands from single-animal pose recovery to the joint problems of detection, segmentation, data association, and long-term identity consistency. One markerless formulation detects heads, tail bases, and body boxes with a deep Part Proposal Network and then solves assignment and part association with a Bayesian Integer Linear Programming model (Jiang et al., 2019). The system introduces the Multi-Mice PartsTrack dataset, consisting of 10 videos, 400 randomly sampled and annotated frames per video, and labels for head, tail base, and body box under behaviors such as approaching, following, nose contact, solitary behavior, and pinning. The BILP combines a motion model, learned geometric constraints, and pairwise part association probabilities, and on the 3-mice dataset its full configuration reaches a MOTA of and an IDF1 of , outperforming SORT, DeepSORT, MOTDT, and related baselines (Jiang et al., 2019).
At mask level, MouseSIS defines space-time instance segmentation as the task of segmenting instances throughout the entire duration of the sensor input using quasi-continuous events and optionally aligned frames (Hamann et al., 2024). The dataset contains 33 sequences of about 20 seconds each, aligned grayscale frames and events, 75,532 annotated masks, and groups of up to seven freely moving and interacting mice. Two reference methods are provided: ModelMixSort, a tracking-by-detection pipeline using E2VID, YOLOv8, SAM, and XMem; and EventSeqFormer, a transformer-based tracking-by-query adaptation of SeqFormer. On the test set, ModelMixSort with frames and events reaches a MOTA of $54.94$, IDF1 of $65.17$, HOTA of $54.19$, and AssA of $10$0, substantially exceeding its frames-only counterpart and illustrating that leveraging event data can consistently improve tracking performance, especially in difficult scenarios (Hamann et al., 2024).
In continuous home-cage monitoring, MouseTracks has also been formalized as a StrongSORT-inspired multi-object tracker embedded in a three-part real-time identification pipeline (Oberhauser et al., 10 Jul 2025). In that setting, mHydra detections are passed to MouseTracks, which combines motion cues from a Kalman filter and box IoU with appearance cues from Mouseformer embeddings, using a weight of $10$1 for motion and $10$2 for appearance. Tracklets are then labeled by MouseMap, a constraint-based linear program that assigns final identities based on ear-tag classifier confidence. The full system runs at 30 frames per second with 24/7 cage coverage and, on 100 minutes of held-out DAX3 video containing 179,980 frames and 524,663 detections, achieves IDF1 $10$3, MOTA $10$4, $10$5 ID switches, and $10$6 correct ID assignment, improving on detector-tracker combinations based on SuperAnimal, DeepLabCut, and SLEAP (Oberhauser et al., 10 Jul 2025).
4. Rodent biomechanics, markers, and high-speed tracking
Before the recent emphasis on markerless deep models, MouseTracks-type systems in rodent biomechanics often referred to high-throughput tracking of painted anatomical markers. A representative example compares SLIC superpixels segmentation with hue thresholding in treadmill recordings of six Sprague–Dawley rats captured by four side-view cameras at 250 frames per second (Maghsoudi et al., 2017). Each rat carried five painted markers—toe, ankle, knee, hip, and anterior superior iliac spine—and the objective was automatic segmentation and tracking suitable for subsequent Direct Linear Transform reconstruction and kinematic analysis.
The SLIC-based method groups pixels using both color and spatial information and then tracks marker candidates with a weighted function using position, speed, shape, hue, size, and grayscale intensity. The hue-thresholding baseline performs segmentation in HSV space followed by the same tracker. On 900 manually evaluated marker regions, SLIC achieved sensitivity $10$7, specificity $10$8, accuracy $10$9, and precision 0, whereas hue thresholding reached sensitivity 1, specificity 2, accuracy 3, and precision 4. During tracking, hue thresholding produced 574 mistakes, compared with 12 for SLIC, leading the authors to conclude that the SLIC superpixels method was superior because the segmentation was more reliable and based on both color and spatial information (Maghsoudi et al., 2017).
This lineage remains relevant because many current markerless systems still inherit the same downstream demands: high frame rates, preservation of limb-level kinematics, and robustness under rapid locomotor phases. A plausible implication is that biomechanics-oriented MouseTracks systems can be ordered along a spectrum from hand-engineered segmentation and tracking to learned detection, pose, and global optimization, rather than treated as separate methodological traditions.
5. MouseTracks as cursor-based attention and saliency estimation
In video saliency, MouseTracks denotes systems that use mouse movements as a scalable proxy for gaze. A notable formulation introduces a mouse-contingent, foveated video player that runs in the browser and updates in real time, making sharpness depend on cursor position through a two-level Gaussian pyramid (Lyudvichenko et al., 2019). For a pixel 5, the displayed intensity is
6
with 7 and 8, where 9 is the video width. Videos are shown full screen, the participant’s mouse movement freely controls the foveated center, and client-side checks exclude users whose screens are smaller than 1024 px or whose browsers cannot sustain at least 20 FPS.
Mouse trajectories are converted into per-frame saliency maps by Gaussian kernel density estimation,
0
with 1. On 12 videos from Hollywood-2, collected through Subjectify.us from 30 participants and yielding 22–30 views per video, mouse-based maps were compared with eye-tracking maps from 16 observers. The central empirical result is that mouse-tracking data from 2 observers achieves roughly the same quality as saliency maps from 1 eye-tracking observer. A temporal semiautomatic model based on SAM-ResNet, ConvLSTM, and mouse-derived priors reaches a Similarity Score of approximately 2, outperforming raw mouse saliency and also outperforming eye-tracking saliency maps from 8 observers while using a prior corresponding to 3 eye observers or about 10 mouse observers (Lyudvichenko et al., 2019).
The literature emphasizes that the quality of cursor-based saliency depends critically on interface design. Without the foveated player, cursor positions might reflect navigation or idling rather than visual attention. This suggests that MouseTracks in saliency research is not merely passive logging; it is an interaction protocol engineered so that cursor dynamics approximate gaze.
6. Behavioral biometrics, profiling, bot detection, and continuous authentication
In security and HCI, MouseTracks refers to systems that treat cursor behavior as a behavioral biometric. The general survey literature defines mouse dynamics as the pattern of a user’s mouse behaviors on a GUI and widget interactions as the same signal augmented with the identity of the target GUI element, with feature sets spanning path length, elapsed time, velocity, acceleration, jerk, curvature, angular changes, click timing, and Fitts’ Law variables (Khan et al., 2022). The survey also organizes data collection into fixed static tasks, app-restricted continuous tasks, app-agnostic semi-controlled tasks, and completely free data collection, and frames evaluation in terms of FAR, FRR, EER, ROC, and AUC (Khan et al., 2022).
Browser fingerprinting work shows that coarse spatial distributions of mouse movement can be highly identifying even without explicit trajectory modeling. In a small study with 6 users sharing 2 PCs and 2 browsers, browser statistics alone achieved 3 accuracy, mouse heatmaps alone achieved 4, and gaze heatmaps 5. In a larger study with 80 users and 800 sessions, heatmap-based mouse fingerprints reached up to 6 accuracy with a Subspace Discriminant ensemble, and consistently outperformed classical mouse statistics (Fuhl et al., 2021). A related privacy-focused study demonstrated that raw cursor trajectories from 1,467 search sessions can be fed directly to a bidirectional GRU with 64 units to infer age and gender; the model achieved age AUC 7 and gender AUC 8, while an adversarial obfuscation method based on adding synthetic mousemove events with Gaussian noise reduced AUC to around 9 already at 0 px (Leiva et al., 2021).
Continuous authentication research reaches similarly high discriminative performance when short temporal windows are used. In a dataset collected from 19 students playing Team Fortress 2 and Poly Bridge on identical hardware, raw mouse-event streams were converted into 40-event sequences with features including X, Y, Stop Duration, Jerk, Direction Change, Movement Distance, Acceleration, Button Presses, and Angle (Dave et al., 2024). In binary user-vs-others verification, a GRU obtained test AUC 1 on Team Fortress 2, 2 on Poly Bridge, and 3 on the combined dataset, with corresponding test F1 values 4, 5, and 6 (Dave et al., 2024).
Bot detection treats the same signal from an adversarial perspective. BeCAPTCHA-Mouse models human trajectories by a Sigma-Lognormal neuromotor decomposition, constructs a 37-dimensional feature vector from stroke parameters, and augments training with 5,000 function-based and 5,000 GAN-based synthetic trajectories in a 15,000-trajectory benchmark drawn from 58 users (Acien et al., 2020). Its experiments report about 7 average accuracy on highly realistic bot trajectories using only one mouse trajectory, and relative improvement of more than 8 when neuromotor features are fused with state-of-the-art mouse dynamic features (Acien et al., 2020). Across these studies, the recurring pattern is that mouse dynamics are useful precisely because they are low-level, continuous, and difficult to reduce to a single discrete event.
7. Dynamic modeling, limitations, and research directions
Beyond classification pipelines, MouseTracks has also been theorized as a generative time-series object. In experimental psychology, a state-space approach to dynamic modeling of mouse-tracking data critiques the dominant two-step procedure in which trajectories are first reduced to summary statistics and then analyzed with standard models (Calcagnì et al., 2019). The proposed framework converts normalized 9-0 trajectories into angle sequences, introduces a latent Gaussian random walk 1, and uses a two-component von Mises mixture observation model with a logistic link to experimental predictors. Estimation is performed by a Metropolis–Hastings algorithm coupled with a non-linear recursive filter, and posterior predictive fit in a lexical decision case study reached 2 overall for the categorical-factor model, 3 for the bigram-only model, and 4 for the full interaction model (Calcagnì et al., 2019). This formulation makes explicit that many MouseTracks systems can be interpreted as latent-state estimators rather than as mere feature extractors.
Across domains, the main limitations are also recurrent. The multi-view 3D animal-pose framework based on rigid templates, deep-learned deformations, and global smoothness is still a concept, with full implementation and empirical evaluation on real mice pending (Hellwich et al., 19 Jan 2025). The home-cage identity pipeline currently depends on custom ear tags, reports that about 5 of detections have ear tags not visible or fallen off, and is evaluated primarily in three-mouse cages (Oberhauser et al., 10 Jul 2025). MouseSIS is limited to one cage type, one strain, and baselines that rely partly on E2VID, which degrades under low contrast-threshold event noise (Hamann et al., 2024). In browser security, mouse data are freely accessible in the browser and do not have to be activated manually like the camera, which raises obvious privacy implications, especially when cursor traces can support covert tracking or demographic profiling (Fuhl et al., 2021, Leiva et al., 2021). The broader survey literature adds cross-device variability, concept drift, spoofing, and lack of standardized benchmarks to this list (Khan et al., 2022).
These constraints indicate that MouseTracks remains a moving target rather than a closed technical category. In animal vision, likely directions include articulated skeletons, multi-animal pose under heavy occlusion, event-native segmentation architectures, and extensions from three to five mice or more (Hellwich et al., 19 Jan 2025, Hamann et al., 2024, Oberhauser et al., 10 Jul 2025). In HCI and security, the literature points toward richer multimodal fusion, online adaptation, stronger privacy protections, and more principled generative models of human cursor dynamics (Khan et al., 2022, Calcagnì et al., 2019). Across all uses, the central technical challenge remains the same: extracting stable, semantically useful structure from incomplete, noisy, and highly contextual trajectories.
