MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes

Abstract: Modern smart vision sensors need on-device intelligence to process video streams, as cloud computing is often impractical due to bandwidth, latency, and privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited memory and compute, making conventional video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the Transformer, while reducing multiply-accumulate operations by as much as 53\% for the CNNs and 32\% for the Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55\% energy savings compared to processing only full-resolution images, enabling the first real-time Transformer-based VOD on an MCU-class embedded vision node. Code available at https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access

• The paper introduces a causal, training-free VOD pipeline that combines multi-resolution inference, ByteTrack-based tracking, and a probabilistic rescore mechanism.
• It leverages both CNN and lightweight Transformer backbones to achieve near-lossless accuracy while reducing compute by up to 53% and energy by up to 55%.
• Empirical results on the GAP9 MCU demonstrate robust real-time performance and efficient deployment for embedded vision applications.

MR2-ByteTrack: Enabling Efficient VOD with CNNs and Transformers on Embedded MCUs

Motivation and Context

Embedded vision nodes are increasingly central to edge devices in diverse domains—ranging from autonomous robotics to IoT-based surveillance and industrial sensing—where real-time object detection within strict memory and power budgets is required. Traditionally, frame-by-frame DNN inference discards temporal cues in the data stream, leading to diminished robustness under occlusion and noise. State-of-the-art Video Object Detection (VOD) pipelines leverage multi-frame temporal aggregation to enhance accuracy; however, they are memory- and compute-intensive, thus unsuited for the ultra-efficient MCUs prevalent in embedded systems.

MR2-ByteTrack directly addresses these constraints by introducing a strictly causal, training-free VOD pipeline designed for MCU deployment. The method combines multi-resolution inference with a stateful tracker and a novel probabilistic class rescoring mechanism, enabling efficient yet accurate video understanding with both CNN and lightweight Transformer backbones.

Figure 1: Overview of the MR2-ByteTrack algorithm, showing core components including alternating (multi-resolution) object detection, tracking, and the Rescore module.

Methodology

Multi-Resolution Inference

The pipeline interleaves high-resolution (full-res) inferences with a configurable sequence of low-resolution (low-res) inferences. Given a periodicity $P$, one full-res detection is followed by $P$ low-res frames. This yields significant computational savings: for CNNs, MACs scale nearly linearly with input size; for the EffViT-Det Transformer, ReLU-based linear attention enables similar linear scaling, bypassing quadratic attention costs. Importantly, weight sharing ensures no increase in model storage footprint regardless of resolution scheme.

ByteTrack-Based Tracking

At each timestep, bounding box detections (class, confidence) are input to a ByteTrack-inspired tracker using IoU-based Hungarian matching and a lightweight Kalman filter for motion prediction. Active tracks are managed based on confidence thresholds and persistence, ensuring robust object association even as resolution varies. In contrast to conventional ByteTrack, MR2-ByteTrack operates strictly causally (no future frames, no retrospective refinement) and is tailored for multi-class, resolution-variant inference outputs.

Probabilistic Rescore Mechanism

The Rescore algorithm refines tracked object class assignments and confidence scores by integrating temporal evidence, using union and normalized margin update rules to aggregate detection confidences and correct inconsistent class assignments. This mechanism is entirely training-free, dataset-agnostic, and exploits temporal smoothness in object motion and class prediction.

Efficient Transformer-Based Detection: EffViT-Det

Typical ViTs and DETR-type architectures are not MCU-feasible due to compute and memory footprint. The paper leverages EfficientViT-B0 as a backbone, composed of early convolutional blocks and three attention-based layers with ReLU-based linear attention (linear in input area). An FPN is replaced by a higher-performing Path Aggregation Network (PAN) similar to YOLOX, resulting in a hybrid model (EffViT-Det) that supports multi-resolution inference and MCU deployment at $<$1M parameters.

Embedded Hardware Deployment

MR2-ByteTrack is validated on the GAP9 RISC-V MCU, a platform with 1.5 MB L2 SRAM, 128 KB L1, and both octa-SPI RAM/Flash. The pipeline's modularity—splitting between high/low-res binary codes, dynamic overlay in L2 for code execution, and off-chip storage for weights/activations—enables deployment even for Transformer backbones constrained by code size. Minor code loading overheads ($<1$ ms) and trivial ByteTrack/Rescore RAM usage ($<$1% of detector cost) demonstrate real-world efficiency, with multi-resolution operation requiring only a single instantiation of each pre-trained model (no multi-network ensembles).

Figure 2: Overview of the GAP9 MCU and memory hierarchy, showing efficient code/data allocation strategies for on-device inference.

Empirical Results

Comprehensive experiments on ImageNetVID validate the approach with NanoDet, YOLOX (CNNs), and EffViT-Det backbones. For each model:

• Accuracy: MR2-ByteTrack at high frame-rate settings matches or exceeds baseline frame-by-frame models’ mAP, e.g., 55.7 mAP (NanoDet) and 52.4 mAP (EffViT-Det) at $P=0$ full-res. Under multi-resolution, at $P=5$ (NanoDet), MR2-ByteTrack yields only +0.6% mAP relative to baseline, with 53.4% compute reduction.
• Efficiency: Throughput and energy per frame improve up to $2.2\times$ and $2\times$, respectively, versus full-res processing, with negligible overhead from tracker/rescore modules.
• Transformer Deployment: Demonstrated for the first time on an MCU, EffViT-Det at $P=1$ (one low-res per full-res) maintains mAP and F1 but halves compute.
• Precision-Recall Trends: For CNNs, both precision and recall degrade gracefully with increased $P$0, consistent with fixed receptive fields limiting small-object recovery at low-res; for Transformers, recall remains stable, while precision degrades, indicating false positive propagation via tracking.

  Figure 3: mAP (top) and MMAC (bottom) of MR2-ByteTrack and baselines as $P$1 (number of low-res frames between each high-res frame) varies. Marked in red are MR2-ByteTrack configurations achieving baseline or higher accuracy at much lower compute cost.

  

  Figure 4: Precision (top) and recall (bottom) of MR2-ByteTrack and baseline models across different multi-resolution settings.

  

  Figure 5: Power profile of EffViT-Det on GAP9 for full-res and low-res inference, showing code loading and auxiliary operations as negligible contributors to total energy.

Discussion and Implications

MR2-ByteTrack delivers strong empirical claims: lossless or near-lossless accuracy with up to 53% compute reduction (CNN) or 32% (Transformer), and energy savings up to 55%. The causality and training-free deployment distinguish it from prior VOD pipelines, which require feature buffering, retrospective multi-frame aggregation, or bespoke/ensemble architectures—often infeasible on MCUs (2605.15423). The demonstrated generality (single pre-trained model, adaptable resolution, class-agnostic rescoring) and Transformer backbone deployment substantially expand the design space for edge VOD.

The practical implications are substantial: for real-time, ultra-low-power IoT and mobile robots, this enables object detection and continuous video analytics entirely on-node, minimizing cloud offload, maximizing privacy, and greatly lowering system power. On the theoretical front, the use of probabilistic union rules for class aggregation and the efficacy of linear attention in MCUs suggest further research directions for adaptive, resource-informed DNN execution.

Additional future advances could exploit track-predicted spatial regions for selective high-res inference, introduce unsupervised class-prior adaptation into Rescore, and improve tracker motion models for robustness under rapid ego-motion scenarios.

Conclusion

MR2-ByteTrack establishes a causal, resolution-adaptive, and training-free paradigm for VOD on embedded platforms, demonstrably bridging the accuracy/efficiency gap for both CNNs and Transformers under tight MCU constraints. The method’s modularity, hardware compatibility, and precise temporal logic broaden the accessibility of advanced VOD to extreme edge devices, with immediate applicability to resource-constrained AI systems and embedded continuous vision applications.