MouseMap: Multimodal Input Mapping

• MouseMap is a family of mapping techniques that convert spatial and temporal signals (from cameras, clicks, or tracking data) into actionable outputs.
• It encompasses diverse methodologies such as rule-based gesture recognition, spatial trace-to-prior conversion, and constraint programming for global identity assignment.
• Applications span from virtual mouse interfaces and attention mapping in multimodal retrieval to laboratory animal tracking, offering practical insights into trade-offs between simplicity and robustness.

Searching arXiv for papers explicitly mentioning “MouseMap” and closely related usages so the article can be grounded in current arXiv literature. MouseMap denotes, in several arXiv contexts, an explicit mapping from mouse-like input signals to downstream computational outputs. In the vision-based HCI literature, it is the mapping from visual markers or colored regions to cursor movement and mouse events (Kumar et al., 2012, Puri, 2014, Banerjee et al., 2014). In multimodal retrieval and attention modeling, it is the conversion of mouse traces or clicks into spatial priors or importance maps (Changpinyo et al., 2021, Kim et al., 2017). In laboratory animal monitoring, MouseMap is the name of a final tracklet associator that assigns globally consistent animal identities to tracklets (Oberhauser et al., 10 Jul 2025). This suggests that MouseMap is best understood as a family of mappings from cursor, gesture, or trajectory signals to actions, spatial constraints, or identity labels rather than as a single standardized framework.

1. Core meanings and scope

Across the cited literature, MouseMap appears in three technically distinct but structurally related senses. Each sense begins with temporally ordered, spatially localized input and ends with a compact representation that can drive a downstream system.

Context	Input	Output
Vision-based mouse simulation	Colored tapes or colored caps in webcam frames	Cursor movement and mouse events
Multimodal retrieval and attention mapping	Mouse traces or clicks	Spatial priors, bounding boxes, or importance maps
Home-cage monitoring	Tracklets plus per-frame ear-tag predictions	Final ID assignments to tracklets

In "Mouse Simulation Using Two Coloured Tapes" (Kumar et al., 2012), the mapping is from a web camera stream observing yellow tape on the index finger and red tape on the thumb to MoveCursor, LeftClick, RightClick, and DoubleClick. In "Gesture recognition based mouse events" (Puri, 2014), the mapping is rule-based over the number of ROIs and their pixel distance. In "Mouse Control using a Web Camera based on Colour Detection" (Banerjee et al., 2014), the mapping is from the presence of red, green, and blue blobs to pointer motion and click events.

In "Telling the What while Pointing to the Where" (Changpinyo et al., 2021), the query consists of a text sequence $y = (y_1, \ldots, y_K)$ and mouse traces $t$, and the trace is converted into per-utterance 5D vectors $$b = (x_{\min}, x_{\max}, y_{\min}, y_{\max}, \textit{area})$$. In "BubbleView" (Kim et al., 2017), discrete clicks on blurred images are transformed into smoothed click maps that approximate fixation density maps.

In "Towards Continuous Home Cage Monitoring" (Oberhauser et al., 10 Jul 2025), MouseMap is the paper’s named tracklet-to-ID associator / final ID solver. It is not a cursor interface but a global assignment layer: tracklets from MouseTracks and ear-tag confidences from Mouseformer are converted into a single, consistent animal ID for each tracklet.

2. Vision-based virtual mouse systems

The most literal MouseMap appears in webcam-based mouse replacement systems. "Mouse Simulation Using Two Coloured Tapes" (Kumar et al., 2012) defines an architecture of Camera → Image Frame → Background & skin processing → Tape color detection (HSV-based) → Gesture interpretation → Mouse event generation. The system captures a pair of background images at startup, performs background subtraction, applies a histogram-based skin segmentation in RGB space using $P(\text{skin} \mid c)$ and $P(\text{non-skin} \mid c)$, converts the result from RGB to HSV, thresholds for yellow and red, reduces noise with a convolution phase, identifies the largest connected component for each color, and computes centroids $(x_Y, y_Y)$ and $(x_R, y_R)$. Cursor control uses the yellow tape centroid, while click events use the Euclidean distance

$d = \sqrt{(x_Y - x_R)^2 + (y_Y - y_R)^2}$

together with two reference distances, $D$ and $D'$, and a waiting time of 7 seconds. The system adopts absolute position mapping rather than weighted-speed cursor control, and notes that if screen resolution exceeds camera resolution, cursor “jumping” of up to 4 pixels can occur. The paper explicitly implements cursor movement, left click, right click, and double click; scrolling and drag are mentioned in the abstract, but the body only details left/right/double click (Kumar et al., 2012).

"Gesture recognition based mouse events" (Puri, 2014) uses colored caps and a standard 1.3-megapixel laptop webcam, with a MATLAB setup that includes YUY2_320x240. The cap color is selected by a startup snapshot and impixel(), then converted from RGB to YCbCr using

$t$0

and the system thresholds in Cb/Cr space with a value of approximately 12 for green in their setup. The resulting ROIs define a compact gesture vocabulary: 1 ROI for cursor movement, 2 ROIs with distance above threshold for left-click and below threshold for right-click, 3 ROIs for drag, and 4 ROIs for double-click. The paper reports that cursor movement has a recognition rate of approximately 95%, and that a distance threshold of 2 meters is suggested for reliable recognition with the 1.3 MP webcam (Puri, 2014).

"Mouse Control using a Web Camera based on Colour Detection" (Banerjee et al., 2014) uses a standard webcam, MATLAB, and Java integration. The image is flipped, converted to grayscale, and color detection is performed by subtracting individual color channels from the grayscale image, followed by binarization at 20% of the maximum possible value, i.e. $t$1. The centroid of the red region drives the cursor, while red + green triggers a left click and red + blue triggers a right click. This yields a particularly simple MouseMap semantics: red alone for tracking, red-plus-green for left-click, and red-plus-blue for right-click (Banerjee et al., 2014).

Taken together, these systems define MouseMap as an explicit interface layer between low-cost visual sensing and standard GUI semantics. Their common design pattern is marker-based segmentation, centroid extraction, and rule-based event generation. A plausible implication is that “MouseMap” in this subliterature names not a specific classifier but a deterministic transduction scheme from image-space marker configurations to operating-system mouse behavior.

3. Mouse traces as spatial priors and importance maps

A second meaning of MouseMap arises when the mouse is not replacing a pointing device but recording spatial intent. "Telling the What while Pointing to the Where" (Changpinyo et al., 2021) proposes a multimodal query in which the user simultaneously uses both spoken natural language and mouse traces over an empty canvas. The raw trace is a time-ordered sequence

$t$2

and each utterance-aligned segment is converted into a normalized box with spatial padding: $t$3 These box embeddings are processed by a Trace Box Embedder (TBE), concatenated with text token embeddings, and fused by a 6-layer Transformer encoder. The retrieval score becomes $t$4, and training uses the same symmetric batched contrastive loss as the base model. On Flickr30k Localized Narratives test data, the paper reports that without pretraining, text-only retrieval achieves R@1 = 63.5, R@5 = 87.4, R@10 = 92.8, mAP = 74.0, while text+trace achieves R@1 = 68.2, R@5 = 88.8, R@10 = 94.4, mAP = 77.7. Under the best pretraining regime, best text-only reaches R@1 = 83.4, R@5 = 97.6, R@10 = 98.5, mAP = 89.7, whereas best text+trace reaches R@1 = 90.6, R@5 = 98.2, R@10 = 99.4, mAP = 94.0, which the paper reports as approximately 43% relative error rate reduction at R@1 (Changpinyo et al., 2021).

"BubbleView" (Kim et al., 2017) provides a different but related MouseMap: participants inspect a uniformly blurred image by clicking to reveal small circular bubbles at the original resolution. Clicks are collected as coordinates $t$5 and converted into a continuous map by Gaussian smoothing,

$t$6

The paper evaluates 10 experiments with 28 different parameter combinations across information visualizations, natural images, static webpages, graphic designs, and SALICON images. BubbleView reports that clicks can successfully approximate eye fixations and rank image and design elements by importance. For MASSVIS visualizations under the description task, BubbleView accounts for approximately 89–90% of eye fixations; for OSIE natural scenes it accounts for 78–80%; for FiWI web pages it reaches 78–79% under appropriate time and task settings. The paper also states that BubbleView data is cleaner and more consistent than related methodologies that use continuous mouse movements, and that 10–15 participants per image are typically enough to reach 97–98% of the asymptotic limit (Kim et al., 2017).

These works formalize MouseMap as a spatial prior rather than an event interface. In one case the prior is a sequence of utterance-aligned bounding boxes; in the other it is a smoothed click-density map. In both cases, the mouse serves as a low-friction channel for encoding “where,” and the resulting representation becomes a model input rather than a control output.

4. MouseMap as global identity assignment in laboratory-mouse monitoring

A third usage is domain-specific and names a solver rather than a user interface. "Towards Continuous Home Cage Monitoring" (Oberhauser et al., 10 Jul 2025) presents a real-time identification pipeline for group-housed laboratory mice wearing custom ear tags. The pipeline consists of MouseTracks, Mouseformer, and MouseMap. MouseTracks takes detections from the mHydra detector and produces tracklets using motion and appearance cues. Mouseformer is a CoAtNet-based fine-grained classifier that assigns ear-tag class predictions and confidences for five classes: brown-checkered, red-barred, black-all-filled, noread, and no-eartag. MouseMap then aggregates those predictions at the tracklet level and produces the final identity assignment for each tracklet (Oberhauser et al., 10 Jul 2025).

The paper explicitly characterizes MouseMap as a constraint programming algorithm implemented with cpmpy and relates it to the hotel room problem. The stated objective is: “We use the sum of the confidence of each ear tag classifier prediction in a tracklet as our objective function.” Operationally, MouseMap receives the set of tracklets, the sequence of per-frame ear-tag predictions and confidences, and solves a global assignment problem over a finite set of identities, typically three mice. The paper also describes a pre-solver step that selects “the three most probable tracklet hypotheses” and merges shorter tracklet hypotheses using stitching when they overlap in space but not in time (Oberhauser et al., 10 Jul 2025).

The full system assigns animal IDs at 30 frames per second with 24/7 cage coverage. Over 100 minutes of 3-mouse cage video under varied environments, the complete system (mHydra + Envision, which includes MouseMap) achieves IDF1: 88.01, MOTA: 91.92, and # ID switches: 2.06 per minute. Reported comparisons include SuperAnimal + DeepLabCut with IDF1: 72.33, MOTA: 77.21, # Switches: 24.5; SuperAnimal + SLEAP with IDF1: 76.16, MOTA: 84.54, # Switches: 21.89; and mHydra + SLEAP with IDF1: 67.59, MOTA: 91.03, # Switches: 11.94. The paper also reports an overall ID accuracy of 95.28% and notes that for 33% of all objects detected, the ear tags are either not visible or have fallen off completely (Oberhauser et al., 10 Jul 2025).

Here MouseMap no longer refers to a mapping from human gesture to cursor action. It is a global consistency layer that reconciles local tracking evidence into stable per-animal identities. The commonality with the earlier senses lies in structure: temporally indexed spatial evidence is reduced to a concise, usable assignment.

5. Tactile and embodied MouseMaps

MouseMap can also denote a mapping from cursor position to tactile feedback. "Mouse Embedded Soft Vibration Actuator for Exploring Surface Textures" (Kurokawa et al., 2024) proposes SVA-M, an optical mouse with an embedded Soft Vibration Actuator under the user’s index fingertip. The actuator consists of four soft tubes laid in parallel with 1 mm spacing. The operating principle is electromagnetic, with force

$t$7

and in the experiment the actuator is driven by a square-wave current at 120 Hz when active (Kurokawa et al., 2024).

The mapping is direct: cursor on black produces vibration at 120 Hz, and cursor on white produces no vibration. Physical mouse motion of 40 mm on the desk corresponds to 1000 pixels of cursor movement on a 1920×1080, 13.3" screen, i.e. approximately 25 px/mm. The experimental textures are vertical black-and-white stripe patterns with line widths of 1, 2, 4, 8, 16, 32 pixels, corresponding to 0.04 mm, 0.08 mm, 0.16 mm, 0.32 mm, 0.64 mm, and 1.28 mm in real-world distance. The evaluation uses 5 male participants, 19–23 years old, in a Two-Alternative Forced Choice (2AFC) fineness judgment with 120 trials in total (Kurokawa et al., 2024).

The reported conclusion is that the system can identify a fine spatial resolution up to 0.16 mm width, and that the SVA-M can perceive textures with a fineness greater than 0.16 mm. The paper also identifies a sampling limitation: during difficult discriminations the average cursor speed is approximately 240 px/s, and at 60 Hz display refresh that implies 4 px per frame, so very fine stripes can be skipped, producing aliasing and improper vibration switching (Kurokawa et al., 2024).

This tactile usage broadens MouseMap beyond visual interaction. The mouse becomes a bidirectional transducer: physical movement controls the cursor, and digital spatial content is remapped to vibrotactile sensation. A plausible implication is that MouseMap can be treated as a general mapping layer between the spatial state of a cursor and any downstream modality, not only GUI clicks.

6. Limitations, misconceptions, and future directions

A common misconception is that MouseMap names a single established architecture. The literature instead shows several non-equivalent uses: a gesture-to-event mapping (Kumar et al., 2012), a trace-to-spatial-prior mapping (Changpinyo et al., 2021), a click-to-importance-map proxy for visual attention (Kim et al., 2017), a tracklet-to-ID solver (Oberhauser et al., 10 Jul 2025), and a cursor-to-tactile-feedback mapping (Kurokawa et al., 2024). This suggests that MouseMap is best regarded as a functional abstraction whose exact semantics depend on domain.

Another misconception is that all virtual mouse systems fully replicate ordinary mouse behavior. In (Kumar et al., 2012), scrolling and drag are mentioned in the abstract as achievable mouse tasks, but the body only details left click, right click, and double click. The paper also notes dependence on a static background, reasonably stable lighting, and marker visibility. In (Puri, 2014) and (Banerjee et al., 2014), performance depends on color contrast, fixed thresholds, and constrained backgrounds. These limitations indicate that low-cost marker-based MouseMaps trade generality for implementation simplicity.

Mouse traces should likewise not be treated as literal eye-tracking. BubbleView states that it is designed to collect clicks on static images, works best for defined tasks, and produces data that is cleaner and more consistent than continuous mouse movements; it approximates fixations, but it does so through discrete, deliberate actions rather than full scanpath recovery (Kim et al., 2017). The multimodal retrieval model in (Changpinyo et al., 2021) uses coarse per-utterance boxes rather than raw trajectories or precise shapes, and the paper notes sensitivity to noisy traces, large trace boxes, and alignment assumptions.

The term also appears in arXiv records whose technical content is unavailable. For “Hand Gesture Real Time Paint Tool - Box” (Gajjar et al., 2017), the arXiv entry has no PDF or source attached, so there is no system description, no methods section, no equations, algorithms, or results, and no grounded technical MouseMap can be reconstructed from that record alone.

Future directions are domain-specific. For virtual mouse systems, the provided text implies or proposes drag-and-drop, scroll, adaptive color range calibration, multi-hand or multi-finger interactions, 3D tracking, and replacing rule-based thresholds with learning-based methods (Kumar et al., 2012). For multimodal retrieval, the paper suggests interactive search refinement, combining multiple traces and modalities, and applications to image editing, layout generation, and referring expression grounding (Changpinyo et al., 2021). For tactile MouseMaps, the authors propose more complex textures, frequency control based on speed and position, multimodal sensations, and higher temporal sampling to reduce aliasing (Kurokawa et al., 2024). For laboratory mouse monitoring, future work includes scaling to multi-object tracking of five mice and removing the requirement for custom ear tags in favor of tail tattoos or shaving patterns (Oberhauser et al., 10 Jul 2025).

Under these varied formulations, MouseMap consistently denotes an explicit bridge between spatially grounded input and structured output. What changes across domains is not the existence of a map, but the nature of the space being mapped: webcam markers to pointer events, traces to spatial priors, clicks to importance maps, tracklets to identities, or cursor position to tactile sensation.