Exploiting In-Sensor Computing for Energy-Efficient Earth Observation

Abstract: The rapid growth of the satellite industry has driven a significant increase in geospatial data acquisition, highlighting a critical bottleneck: the severe disparity between the volume of collected sensor data and the limited downlink bandwidth available to ground stations. While On-Board Computing (OBC) has helped address this by pre-processing data in orbit, this article further advances the paradigm by introducing an in-sensor computing framework. We present an optimized end-to-end Earth Observation (EO) pipeline tailored for strict computational constraints by integrating TinyML techniques with the Sony IMX500 Intelligent Vision Sensor. Specifically, our approach shifts processing directly to the sensor level, offloading the computation from the primary embedded device, and effectively mitigating the downlink transmission of noisy or irrelevant data. We evaluated several efficient Convolutional Neural Networks (ConvNets), i.e., SqueezeNet, ShuffleNetV2, and MCUNetV1, on the EuroSAT dataset. Experimental results show that, despite the optimizations required for deployment on the IMX500 platform, our models maintain a competitive 96.68% accuracy while operating within its 8 MB constraints. Specifically, the models reach an average processing throughput of 17.40 FPS with a latency of 27.43 ms. Furthermore, our system profile exhibits high energy efficiency, with a low energy footprint of 14.19 mJ per inference and an efficiency rating of 42.26 GMAC/J, demonstrating its viability for in-sensor deployment.

• The paper demonstrates a fully-optimized in-sensor deep learning pipeline on the Sony IMX500, enabling on-board inference and bypassing traditional data offloads.
• It employs SqueezeNet, ShuffleNetV2, and MCUNetV1 with aggressive post-training quantization to meet an 8 MB RAM limit while maintaining ~97% classification accuracy.
• Results show low latency (27.43 ms/inference) and energy efficiency (14.19 mJ/inference), significantly mitigating downlink bottlenecks in CubeSat EO missions.

Exploiting In-Sensor Computing for Energy-Efficient Earth Observation

Introduction

This work formalizes a systematic approach for addressing on-board processing constraints in CubeSat-class Earth Observation (EO) missions through the development and validation of a fully-optimized in-sensor deep learning pipeline. By deploying TinyML-class convolutional models directly on the Sony IMX500 Intelligent Vision Sensor, the proposed pipeline bypasses the traditional raw data offload to the On-Board Computer (OBC) and drastically mitigates downlink bottlenecks and internal data bus overhead. The empirical investigation centers on deploying SqueezeNet, ShuffleNetV2, and MCUNetV1 to the IMX500 architecture, with extensive quantization and model adjustments to suit the sensor's 8 MB RAM constraint.

Motivation and Context

Recent advances in satellite miniaturization and the proliferation of CubeSats have underscored a severe imbalance in EO workflow design: the raw data generated by high-resolution sensors far exceeds downlink capacity and local compute availability, making energy- and communication-efficient processing essential. While conventional OBC- or accelerator-based pipelines reduce ground station data transfer, they still demand power-intensive internal transfers and impose significant load on constrained system resources.

The work addresses the fundamental limitations of these paradigms by exploiting in-sensor computing, wherein the sensor’s integrated NN accelerator carries out lightweight, early-stage inference, directly outputting high-level semantic data without exporting raw imagery. This paradigm eliminates redundant transfers and substantially lowers per-inference energy budgets. The chosen models offer a disciplined balance of spatial expressiveness, parameter efficiency, and hardware compliance, making them suitable for highly resource-constrained sensing platforms.

Methodology

Hardware Platform and Deployment Pipeline

The Sony IMX500 intelligent vision sensor, a 12.3-megapixel CMOS platform featuring an integrated NN accelerator and on-chip memory, is used as the target architecture. Model deployment is orchestrated through Sony’s Model Compression Toolkit (MCT), which performs static graph tracing and post-training quantization (PTQ) to reduce model representations from FP32 to INT8. The post-quantization models are further refined to conform with platform-specific execution requirements, including memory occupancy and NN engine operation support.

The experimental setup utilizes a Raspberry Pi AI Camera interfaced with a Raspberry Pi 5 as a reference OBC. Following in-sensor computation, only semantic metadata (classifications, ROIs) is offloaded, which simulates a realistic satellite workflow while quantifying processing, latency, and energy metrics in an isolated manner.

Model Selection and Optimization

Chosen architectures—SqueezeNet, ShuffleNetV2, MCUNetV1—are selected for their convolution-centric, hardware-friendly operations, and strong memory/throughput characteristics:

• SqueezeNet: employs Fire modules for aggressive model compression while preserving representational capacity.
• ShuffleNetV2: leverages channel shuffling and pointwise group convolution for improved throughput per computation budget.
• MCUNetV1: optimally co-designed for ultra-constrained MCUs, combining efficient macro-architecture selection and depthwise-separable convolutions.

All models are trained on the EuroSAT dataset with an 80/20 train-test split, using Adam optimizer and standardized hyperparameters.

Empirical Results

Classification, Quantization, and Efficiency

Figure 1: Performance comparison of ShuffleNetV2, MCUNetV1, and SqueezeNet—in terms of accuracy for FP32 and INT8 variants (left), and F1-score, Recall, and Precision for vanilla models (right).

Classification results establish MCUNetV1 as the most effective, achieving 97.76% accuracy, with ShuffleNetV2 close behind at 97.09%. SqueezeNet performs slightly lower at 96.07%. F1-score, recall, and precision metrics confirm balanced and high-confidence classification for all candidates, with MCUNetV1 yielding an F1 of 97.7%.

Quantization analysis reveals minimal loss: ShuffleNetV2’s accuracy declines by only 0.03% after PTQ, while MCUNetV1 exhibits a 0.07% accuracy gain post-quantization, suggesting quantization-induced beneficial regularization. SqueezeNet demonstrates higher sensitivity to quantization (0.92% drop).

All models operate within 8 MB of memory, maintain a mean throughput of 17.40 FPS, and achieve low-latency performance (mean 27.43 ms/inference). Average per-inference energy consumption is 14.19 mJ, with a model-level operational efficiency of 42.26 GMAC/J. The architecture thus supports sustained on-orbit operation under CubeSat-class budgets.

Discussion and Implications

This work demonstrates that in-sensor deep learning—implemented with hardware-aware TinyML optimizations and aggressive PTQ—can deliver on-par classification performance while confining all computation to the sensor die. The energy and communication savings are substantial, as only semantic content ever traverses the main system bus or is downlinked. The deployment strategy directly addresses critical bottlenecks of contemporary OBC-centric architectures, offering substantial improvements in system scalability, autonomy, and operational sustainability.

Practical implications include enabling time- and energy-critical autonomous decision-making (e.g., rapid hazard detection, on-orbit data triaging), prolonging mission lifetimes, and supporting more complex multi-satellite constellations without incurring data bus or communication penalties. Future developments may incorporate on-orbit adaptive learning and robustness validation to spaceborne hazards, further extending the functional lifespan and autonomy of CubeSat EO platforms.

Conclusion

Pushing the deep learning pipeline to the sensor edge, this in-sensor TinyML methodology delivers competitive classification accuracy, strict memory and latency adherence, and significant energy gains under the constraints of spaceborne EO platforms. It represents a formally validated architectural alternative to the power- and bandwidth-limited OBC-based designs. Subsequent research should address radiation hardening, on-orbit incremental learning schemes, and the generalization of in-sensor models to other sensing modalities and tasks.

---

Reference: "Exploiting In-Sensor Computing for Energy-Efficient Earth Observation" (2606.01271)