Mouseformer: Ear-Tag Transformer Classifier

• The paper demonstrates that Mouseformer delivers per-frame fine-grained ear-tag classification and appearance embeddings, facilitating robust identity tracking in challenging home cage settings.
• It uses a CoAtNet backbone with a five-class output scheme to effectively distinguish between visible tags and failure cases (noread, no-eartag), improving recognition reliability.
• Mouseformer integrates with MouseTracks and MouseMap to combine temporal aggregation with appearance matching, resulting in high overall system ID accuracy.

Searching arXiv for the target paper and a few related references for minimal supporting context. Mouseformer is a transformer-based ear-tag identification classifier introduced as the middle stage of a three-part system for continuous home cage monitoring of laboratory mice. In the formulation reported in "Towards Continuous Home Cage Monitoring: An Evaluation of Tracking and Identification Strategies for Laboratory Mice" (Oberhauser et al., 10 Jul 2025), the end-to-end pipeline consists of MouseTracks, Mouseformer, and MouseMap. MouseTracks links detector outputs into short, temporally coherent sequences; Mouseformer classifies each detected mouse crop into one of five ear-tag-related classes while also emitting an appearance embedding; and MouseMap consolidates per-frame identification evidence into a final one-identity-per-tracklet assignment under cage-level constraints. Within that pipeline, Mouseformer is the component responsible for per-frame fine-grained recognition of custom RapID ear tags under occlusion, missing tags, rapid motion, and dense social interaction (Oberhauser et al., 10 Jul 2025).

1. Definition and role within the Envision pipeline

Mouseformer is defined as a transformer-based ID classifier that operates on RGB image crops of mice generated from detector bounding boxes. Its immediate input is an RGB crop from the detector mHydra, and its output is a probability distribution over five classes together with an intermediate appearance embedding. The five classes are: brown-checkered, red-barred, black-all-filled, noread, and no-eartag. In this formulation, the classifier is not limited to assigning one of the visible ear-tag classes; it must also model uncertainty and failure cases, specifically unreadable tags and cases in which the ear tag has fallen off (Oberhauser et al., 10 Jul 2025).

The system-level role of Mouseformer is dual. First, it provides per-frame fine-grained classification of ear-tag appearance. Second, it exposes an appearance representation used by MouseTracks for appearance matching during online tracking. This makes Mouseformer both an identification module and a source of Re-ID-like descriptors for track association. MouseMap then aggregates the per-frame class predictions across time to produce a per-tracklet identity that is consistent with cage-level exclusivity constraints, so Mouseformer is best understood as the local evidence generator within a temporally and combinatorially constrained identification system (Oberhauser et al., 10 Jul 2025).

This design suggests that Mouseformer is not intended to solve identity assignment in isolation. Its predictions are explicitly embedded in a broader pipeline in which tracking continuity, appearance similarity, and tracklet-level optimization jointly determine final identity labels.

2. Input representation, class structure, and ear-tag semantics

The discriminative signal used by Mouseformer is a small patch corresponding to a custom RapID barcoded ear tag. The paper emphasizes that this signal is often difficult to recover because mice are densely housed, visually similar, highly mobile, and frequently occlude one another. Each mouse wears two ear tags, one in each ear, positioned anteriorly and posteriorly. This increases the likelihood that at least one tag is visible in a given frame, but does not remove the need to handle many frames in which neither tag can be read reliably (Oberhauser et al., 10 Jul 2025).

The tag classes used for triply housed mice are tied to custom RapID ear tags with four-square binary patterns over color-coded backgrounds. The three tag classes used in the experiments are red background with a barred pattern, black background with all squares filled, and brown background with a checkered pattern. Mouseformer maps per-frame crops into a five-way class space that augments these three visible tag classes with noread and no-eartag. The noread category captures occluded or unreadable tags, whereas no-eartag captures cases in which the tag has fallen off (Oberhauser et al., 10 Jul 2025).

This class design is structurally important. Rather than forcing all difficult frames into one of the visible tag classes, Mouseformer has explicit failure classes. A plausible implication is that the classifier can represent ambiguity without contaminating the visible-tag posteriors, allowing the downstream MouseMap stage to discount weak or uninformative frames during temporal aggregation.

3. Architecture and model formulation

Mouseformer uses a CoAtNet backbone pretrained on ImageNet-21k. In the reported implementation, CoAtNet functions as a hybrid of convolution and attention and is trained for fine-grained ear-tag classification on mouse crops. The classifier head maps the final backbone features to five class logits. In addition, the model exposes an appearance embedding, described as “a set of vectors for each animal crop,” which is used by MouseTracks for appearance matching (Oberhauser et al., 10 Jul 2025).

The paper does not specify the exact CoAtNet variant, including depth, number of heads, channel sizes, MLP dimensions, number of stages, tokenization details, positional encoding scheme, or the size of the exported appearance embedding. It also does not report temporal tokens, temporal transformers, or cross-attention with motion features. Mouseformer therefore operates as a per-frame image classifier rather than as an explicitly temporal transformer (Oberhauser et al., 10 Jul 2025).

The background attention formulation given in the paper is standard scaled dot-product attention:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

and multi-head attention is described as

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$

with

$$\mathrm{head}_h=\mathrm{Attention}(XW_Q^h,XW_K^h,XW_V^h).$$

The paper notes that normalization and residual connections typically follow pre/post-norm transformer blocks, but exact placements for Mouseformer are not reported beyond the use of CoAtNet (Oberhauser et al., 10 Jul 2025).

The rationale given for a transformer-based classifier is that fine-grained recognition often benefits from attention mechanisms that can focus on small, discriminative regions amid background clutter, pose variation, and occlusion. The paper also states that CoAtNet’s hybrid conv-attention design provides locality bias and translational robustness from convolutions while retaining the global receptive field and dynamic weighting of attention. No direct comparison with pure ViT or Swin is reported (Oberhauser et al., 10 Jul 2025).

4. Data, training regime, and operational conditions

The training set for Mouseformer contains 86,513 mouse crops sampled from the Envision platform, which records cages 24/7 at 30 FPS. The class distribution is brown-checkered at 23%, red-barred at 21%, black-all-filled at 23%, noread at 17%, and no-eartag at 16%. Two hardware versions, DAX2 and DAX3, were used for training, and the reported evaluations use DAX3 test data. The paper states that training with both DAX2 and DAX3 images improved generalization to DAX3-only test data, consistent with attention models benefiting from larger and more diverse datasets (Oberhauser et al., 10 Jul 2025).

For end-to-end validation and testing, the paper reports 100 minutes of video from DAX3 cages, comprising 179,980 frames and 524,663 objects, with 3 mice per cage. Conditions vary across mouse strains, bedding, enrichment, nest materials, and lighting. The reported strains are C3HJ/HeJ, C57BL/6J, and BALB/cJ; bedding conditions include Alpha-dri, Aspen Chips, and Sani-chips; enrichment includes wood blocks; nesting materials include 4–8 g brown shredded paper and 1–2 cotton squares; and lighting varies across light/dark cycles (Oberhauser et al., 10 Jul 2025).

Several implementation details remain unspecified. Crops are taken automatically from mHydra detector bounding boxes, but crop resolution, resizing policy, normalization, and color preprocessing are not specified. Data augmentation is also not specified: the paper does not detail rotations, blur, color jitter, lighting augmentation, or occlusion simulation. Loss function details, class weighting, label smoothing, regularization, optimizer, schedule, batch size, epoch count, training time, and hardware are likewise not reported. The paper notes that a standard choice would be cross-entropy over the five classes, but it does not explicitly state that this was used (Oberhauser et al., 10 Jul 2025).

These omissions are relevant for reproducibility. The paper provides the structural integration of Mouseformer into the pipeline and the empirical outcomes of that integration, but leaves many training and systems parameters to empirical selection.

5. Integration with MouseTracks and MouseMap

Mouseformer’s outputs are consumed by both upstream and downstream components of the full system. MouseTracks merges detections into tracklets by combining motion and appearance costs and solving an assignment per frame using the Hungarian algorithm. The motion model is a SORT-style Kalman filter on bounding boxes, with motion cost derived from IoU between predicted and observed boxes. Detection confidence modulates the Kalman update gain in a StrongSORT-style manner, giving more weight to high-confidence detections (Oberhauser et al., 10 Jul 2025).

The appearance model in MouseTracks is built directly from Mouseformer’s embedding. For each active tracklet, MouseTracks maintains an exponential moving average of the appearance vector:

$e_t=\alpha e_{t-1}+(1-\alpha)f_t,$

where $f_t$ is the L2-normalized Mouseformer embedding at time $t$, and $\alpha$ is not specified. Appearance similarity for association is cosine similarity:

$$\mathrm{sim}(i,j)=\frac{f_i\cdot e_j}{\lVert f_i\rVert \lVert e_j\rVert}.$$

The fused cost is a weighted combination of motion and appearance costs with $\lambda=0.9$ for motion and $1-\lambda=0.1$ for appearance, tuned for mice. A typical fusion reported in the paper is

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$0

where the appearance term can be implemented as cosine distance (Oberhauser et al., 10 Jul 2025).

MouseMap performs tracklet-to-identity assignment as a constraint program. Its goal is to assign each tracklet a single identity from the three cage-specific IDs while maximizing a global consistency objective under mutual-exclusion constraints. The paper states that the solver uses “the sum of the confidence of each ear tag classifier prediction in a tracklet” as the objective. A faithful formalization reported in the paper defines

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$1

where $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$2 is the score for assigning tracklet $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$3 to identity $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$4 and $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$5 is the class probability at frame $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$6. With binary variables $$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$7, the optimization is

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$8

subject to

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_H)W_O,$$9

and, for temporally overlapping tracklets,

$$\mathrm{head}_h=\mathrm{Attention}(XW_Q^h,XW_K^h,XW_V^h).$$0

The paper names cpmpy as the modeling tool and describes this constraint structure using a “hotel room problem” analogy. It also reports a pre-solver pruning step to keep the three most probable tracklet hypotheses and to stitch or merge short hypotheses that do not overlap in time, restoring feasibility in edge cases (Oberhauser et al., 10 Jul 2025).

Within this architecture, Mouseformer supplies two distinct forms of evidence: per-frame class posteriors for identity inference and appearance embeddings for local data association. The system is therefore neither a purely discriminative classifier nor a purely metric-learning model; it is a hybrid component embedded in a multi-stage inference graph.

6. Performance, robustness, and reported empirical behavior

The full pipeline operates at 30 FPS with 24/7 cage coverage. The paper does not report per-frame latency for Mouseformer alone, memory footprint, or exact deployment hardware for inference. On the 100-minute DAX3 validation set with three mice per cage and diverse environmental conditions, the overall system achieves per-frame ID accuracy of 95.28% (Oberhauser et al., 10 Jul 2025).

The end-to-end tracking-and-ID comparison reported in the paper evaluates detector-plus-tracker configurations on the same 100-minute dataset.

System	IDF1	MOTA
mHydra + Envision	88.01	91.92
mHydra + SLEAP	67.59	91.03
SuperAnimal + SLEAP	76.16	84.54
SuperAnimal + DeepLabCut	72.33	77.21

The corresponding ID switches per minute are 2.06 for mHydra + Envision, 11.94 for mHydra + SLEAP, 21.89 for SuperAnimal + SLEAP, and 24.5 for SuperAnimal + DeepLabCut (Oberhauser et al., 10 Jul 2025).

The paper’s interpretation is that combining mHydra with the Envision pipeline yields substantially higher IDF1 and far fewer ID switches than competing methods, indicating better identity stability. MOTA is also highest in the reported comparison. It further states that using Mouseformer embeddings for appearance matching improves tracklet continuity and reduces ID switches. Robustness is demonstrated qualitatively across strains, bedding types, enrichment, nesting materials, and the light/dark cycle, although the paper does not provide numerical ablations for transformer depth, embedding size, or augmentation schemes (Oberhauser et al., 10 Jul 2025).

Training Mouseformer on the combined DAX2+DAX3 dataset improves classification on DAX3-only test data. This suggests sensitivity to imaging domain and indicates that camera heterogeneity in training contributes to deployment robustness (Oberhauser et al., 10 Jul 2025).

7. Limitations, failure modes, and relation to prior approaches

A central limitation is that ear tags are frequently unavailable as usable visual evidence. The paper states that in about 33% of detected objects, ear tags are not visible or have fallen off. In such cases, per-frame predictions fall into noread or no-eartag, which can reduce instantaneous confidence. MouseMap mitigates this by integrating evidence across time rather than relying on single-frame decisions (Oberhauser et al., 10 Jul 2025).

Fast motion and motion blur are not quantified, but the paper notes that they likely contribute to noread predictions and more uncertain embeddings. The two-tag-per-mouse design is presented as a practical mitigation because it increases the probability that at least one tag is visible from the camera. Environmental variability, including bedding, nesting, and lighting changes during dark cycles, increases background clutter. The paper does not quantify degradation due to tag aging or wear-and-tear, and it does not report generalization to entirely new camera models or different cage geometries. The improvement from mixed DAX2+DAX3 training nevertheless suggests that camera-domain variation is a meaningful factor (Oberhauser et al., 10 Jul 2025).

In relation to prior work, the paper states that Re-ID via generic appearance embeddings, such as CNN-based person Re-ID, was considered but was less accurate in this setting than a dedicated ear-tag classifier combined with physical tags. Mouseformer is therefore tailored to a discrete tag vocabulary while still providing an appearance embedding optimized for the domain. The paper does not report backbone comparisons among CoAtNet, ViT, or Swin; CoAtNet is chosen on the basis of strong performance reported in the literature for fine-grained image recognition and scalability (Oberhauser et al., 10 Jul 2025).

A common misconception would be to treat Mouseformer as a complete identity solution on its own. The reported system does not support that interpretation. Mouseformer is a per-frame classifier with an appearance head; stable final identity assignment emerges only from its integration with MouseTracks and MouseMap. Another misconception would be to read the reported accuracy as evidence that tags are visible most of the time. The paper explicitly reports a substantial fraction of detections in which tags are unreadable or absent, and its architecture is built around that fact (Oberhauser et al., 10 Jul 2025).