MST: Multi-State Efficient Object Tracker

• Multi-State Tracker (MST) is an efficient single-object tracking framework that uses a transformer backbone with multi-state specialization to improve feature representation.
• It combines Multi-State Generation (MSG), State-Specific Enhancement (SSE), and Cross-State Interaction (CSI) to balance accuracy and computational efficiency.
• MST achieves robust, real-time performance (up to 200 FPS on GPU) while keeping additional computational overhead and parameter increases minimal.

Searching arXiv for the primary MST paper and closely related efficient tracking context. Using arXiv search tool to retrieve the Multi-State Tracker paper. Multi-State Tracker (MST) is the efficient single-object tracking framework introduced in "Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction" (Wang et al., 15 Aug 2025). It is designed around the observation that efficient trackers achieve faster runtime by reducing computational complexity and model parameters, but that this efficiency often compromises the expense of weakened feature representation capacity, thus limiting the ability to accurately capture target states using single-layer features. MST addresses that limitation by combining Multi-State Generation (MSG), State-Specific Enhancement (SSE), and Cross-State Interaction (CSI) within a one-stream transformer-style pipeline, with the stated aim of improving tracking accuracy and robustness while incurring minimal computational overhead (Wang et al., 15 Aug 2025). This usage should be distinguished from the MST acronym in Multihypothesis Segmentation Tracking (Price et al., 2021) and in the Multi-State Trajectory approach to non-adiabatic dynamics (Tao, 2017).

1. Problem setting and design objective

MST is positioned in the efficient or lightweight tracking regime. The core problem statement is that reduced model size and reduced computational complexity can weaken feature representation, especially when target state estimation relies on single-layer features. The framework therefore seeks to enhance feature representation in an interactive and adaptive manner while preserving runtime efficiency, with the paper emphasizing robustness in complex environments and real-time deployment characteristics (Wang et al., 15 Aug 2025).

The architecture is organized around “multi-state” features rather than a single terminal representation. In the paper’s formulation, MSG generates multiple state representations at multiple stages during feature extraction, SSE refines them to highlight target-specific features, and CSI facilitates information exchange between these states and ensures the integration of complementary features. A plausible implication is that MST treats inter-layer diversity as a structured source of target-state evidence rather than as a by-product of a backbone network.

2. One-stream architecture and data flow

MST adopts a one-stream transformer-style backbone, specifically ViT-Tiny, and inserts three lightweight state-aware modules—MSG, SSE, and CSI—between the backbone and a standard center-head (Wang et al., 15 Aug 2025). The reported high-level data flow is:

1. Patch-embed template $Z$ and search region $X$.
2. Run ViT-Tiny blocks; at the last three layers extract token sets $S^{(L-2)}, S^{(L-1)}, S^{(L)}$ as the MSG stage.
3. Independently refine each state via SSE.
4. Fuse the three enhanced features via CSI.
5. Send the final fused feature $Y$ to a center-head for classification and regression.

Module	Role	Reported effect
MSG	Extracts $S^{(L-2)}, S^{(L-1)}, S^{(L)}$	Produces diverse state-representations
SSE	Refines each state independently	Highlights target-specific features
CSI	Fuses enhanced states adaptively	Integrates complementary features

At the token level, inputs are patch-embedded as

$p_i = W_p v_i + b_p,$

with $W_p \in \mathbb{R}^{d' \times d}$ and $b_p \in \mathbb{R}^{d'}$. The template and search tokens are concatenated and passed through $L$ standard transformer blocks. MSG then extracts three state-feature sets,

$$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$

These are described as “state candidates” capturing coarse–fine contextual information (Wang et al., 15 Aug 2025).

3. MSG, SSE, and CSI as the core mechanism

MSG is architecturally simple: it “taps” the output tokens of the last three transformer layers as multi-state features. The novelty in MST lies less in the extraction itself than in the subsequent specialization and interaction stages. This suggests that the framework treats late-layer transformer outputs as a compact multi-state basis for downstream target modeling rather than as redundant features.

Each state $X$0 is passed through an independent SSE module. Internally, SSE is built on the Hidden State Adaptation SSD (HSA-SSD) block. The description given in the paper has three parts (Wang et al., 15 Aug 2025). First, a dynamic $X$1 convolution projects the state into hidden matrices $X$2 through an attention-driven combination of basis kernels: $X$3 Second, a depth-wise $X$4 convolution acts on $X$5: $X$6 Third, the module discretizes $X$7 and $X$8 via learnable thresholds $X$9, computes a hidden input, applies projection and gating, and produces the enhanced state $S^{(L-2)}, S^{(L-1)}, S^{(L)}$0. The paper characterizes SSE as providing bidirectional global modeling on each state with linear complexity in $S^{(L-2)}, S^{(L-1)}, S^{(L)}$1 and channels.

CSI operates after SSE. The three enhanced states $S^{(L-2)}, S^{(L-1)}, S^{(L)}$2 are concatenated along the token dimension,

$S^{(L-2)}, S^{(L-1)}, S^{(L)}$3

and CSI then reuses the HSA-SSD design to perform bidirectional cross-state fusion. The fused output is split back into three parts and summed elementwise to form the final feature $S^{(L-2)}, S^{(L-1)}, S^{(L)}$4. In the paper’s terminology, this adaptively aggregates complementary cues across scales (Wang et al., 15 Aug 2025).

4. HSA-SSD and computational profile

HSA-SSD is presented as an improved Hidden State Mixer SSD tailored for multi-state features. The paper contrasts it with two reference designs: Bi-directional SSM, with $S^{(L-2)}, S^{(L-1)}, S^{(L)}$5 FLOPs and very sequential execution, and HSM-SSD, with $S^{(L-2)}, S^{(L-1)}, S^{(L)}$6 roughly. HSA-SSD is reported to have the same order as HSM-SSD but with dynamic kernel adaptation (Wang et al., 15 Aug 2025).

Component	FLOPs (G)	Params (M)
Backbone (ViT-T)	1.75	5.49
SSE module (3×)	0.05	0.49
CSI module	0.05	0.17
Head	0.53	2.31
Total MST	2.38	8.46

The two state-space modules therefore add only $S^{(L-2)}, S^{(L-1)}, S^{(L)}$7 GFLOPs and $S^{(L-2)}, S^{(L-1)}, S^{(L)}$8 M parameters. The paper describes this overhead as effectively negligible (Wang et al., 15 Aug 2025). In contextual terms, this is the central systems claim of MST: the tracker attributes its accuracy gains to state specialization and cross-state interaction without abandoning the resource profile expected of efficient trackers.

5. Training objective, prediction head, and inference rule

Training uses mixtures of LaSOT, TrackingNet, GOT-10K (strict split), and COCO, with AdamW for $S^{(L-2)}, S^{(L-1)}, S^{(L)}$9 epochs and $Y$0 epochs on GOT-10K; the batch size is tuned for $Y$1 2080Ti (Wang et al., 15 Aug 2025). The tracking head consists of three parallel convolution branches predicting:

• a classification map $Y$2,
• a size map $Y$3,
• an offset map $Y$4.

The predicted position is

$Y$5

and the predicted bounding box is

$Y$6

The loss is reported as

$Y$7

with $Y$8 and $Y$9. The classification term is Weighted Focal Loss, and the regression terms are $S^{(L-2)}, S^{(L-1)}, S^{(L)}$0 and GIoU. During inference, the classification response is multiplied by a Hanning window, the maximum is selected, and the head offsets are applied (Wang et al., 15 Aug 2025).

6. Benchmarks, ablations, and deployment

The reported benchmark results place MST ahead of prior efficient or lightweight trackers across multiple datasets (Wang et al., 15 Aug 2025). In the lightweight group on GOT-10K, TrackingNet, and LaSOT, MST is reported with AO $S^{(L-2)}, S^{(L-1)}, S^{(L)}$1, SR0.5 $S^{(L-2)}, S^{(L-1)}, S^{(L)}$2, SR0.75 $S^{(L-2)}, S^{(L-1)}, S^{(L)}$3, AUC $S^{(L-2)}, S^{(L-1)}, S^{(L)}$4, Pnorm $S^{(L-2)}, S^{(L-1)}, S^{(L)}$5, P $S^{(L-2)}, S^{(L-1)}, S^{(L)}$6, and $S^{(L-2)}, S^{(L-1)}, S^{(L)}$7 FPS on GPU/CPU. The comparison rows reproduced in the paper include HCAT and HiT-Base. On TNL2K, UAV123, NFS, and LaSOT$S^{(L-2)}, S^{(L-1)}, S^{(L)}$8, MST is reported with AUC values $S^{(L-2)}, S^{(L-1)}, S^{(L)}$9, $p_i = W_p v_i + b_p,$0, $p_i = W_p v_i + b_p,$1, and $p_i = W_p v_i + b_p,$2, respectively. The paper summarizes these results by stating that across all datasets MST leads by $p_i = W_p v_i + b_p,$3–$p_i = W_p v_i + b_p,$4 points, and that on GOT-10K it outperforms HCAT by $p_i = W_p v_i + b_p,$5 AO.

The ablation studies isolate the roles of MSG, SSE, and CSI. Removing SSE yields $p_i = W_p v_i + b_p,$6 GOT-10K AO and $p_i = W_p v_i + b_p,$7 UAV123 AUC; removing CSI yields $p_i = W_p v_i + b_p,$8 and $p_i = W_p v_i + b_p,$9; removing both SSE and CSI yields $W_p \in \mathbb{R}^{d' \times d}$0 and $W_p \in \mathbb{R}^{d' \times d}$1; removing MSG, SSE, and CSI yields $W_p \in \mathbb{R}^{d' \times d}$2 and $W_p \in \mathbb{R}^{d' \times d}$3. A separate state-count study reports that three states perform best: one state gives $W_p \in \mathbb{R}^{d' \times d}$4 GOT-10K and $W_p \in \mathbb{R}^{d' \times d}$5 UAV123, two states give $W_p \in \mathbb{R}^{d' \times d}$6 and $W_p \in \mathbb{R}^{d' \times d}$7, three states give $W_p \in \mathbb{R}^{d' \times d}$8 and $W_p \in \mathbb{R}^{d' \times d}$9, and four states give $b_p \in \mathbb{R}^{d'}$0 and $b_p \in \mathbb{R}^{d'}$1. Within the paper’s experimental frame, this identifies the three-state design as the best trade-off among the tested alternatives.

The SSE/CSI backbone-variant comparison further attributes the gains to HSA-SSD. The baseline without SSE/CSI reports $b_p \in \mathbb{R}^{d'}$2 GOT-10K and $b_p \in \mathbb{R}^{d'}$3 UAV123 at $b_p \in \mathbb{R}^{d'}$4 GFLOPs, $b_p \in \mathbb{R}^{d'}$5 M parameters, and $b_p \in \mathbb{R}^{d'}$6 FPS. Bi-SSM reports $b_p \in \mathbb{R}^{d'}$7 and $b_p \in \mathbb{R}^{d'}$8 at $b_p \in \mathbb{R}^{d'}$9 GFLOPs, $L$0 M, and $L$1 FPS; HSM-SSD reports $L$2 and $L$3 at $L$4 GFLOPs, $L$5 M, and $L$6 FPS; HSA-SSD reports $L$7 and $L$8 at $L$9 GFLOPs, $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$0 M, and $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$1 FPS (Wang et al., 15 Aug 2025).

For deployment, the paper reports real-time performance on GPU at $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$2 FPS and on CPU at $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$3 FPS, and $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$4 FPS on NVIDIA Jetson Xavier NX with GOT-10K AO $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$5. The comparison table reproduced in the paper lists HiT-Tiny at $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$6 FPS and AO $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$7, HiT-Small at $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$8 FPS and AO $$S^{(L-2)},\; S^{(L-1)},\; S^{(L)}, \qquad S^{(i)} \in \mathbb{R}^{B \times (M+N) \times d}.$$9, HiT-Base at $X$00 FPS and AO $X$01, and MST at $X$02 FPS and AO $X$03. The stated scenarios include UAV video, robotics (SLAM), AR, and surveillance on resource-poor platforms. A plausible implication is that MST is intended not merely as an accuracy-improving backbone modification, but as a practical efficient-tracking design for edge-constrained operation.