PicoSAM2: Lightweight In-Sensor Segmentation

• PicoSAM2 is a lightweight, promptable segmentation model designed for on-sensor deployment using fixed-point prompt encoding to eliminate runtime overhead.
• It employs a depthwise separable U-Net architecture with INT8 quantization, achieving 51.9% mIoU on COCO and 44.9% mIoU on LVIS, tailored for the Sony IMX500.
• The model delivers privacy-preserving, real-time segmentation within strict memory and operator constraints by processing RGB input directly on-sensor.

Searching arXiv for PicoSAM2 and closely related papers to ground the article in current literature. PicoSAM2 is a lightweight, promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500. It is described as a depthwise separable U-Net distilled from Segment Anything Model 2 (SAM2), with fixed-point prompt encoding that eliminates runtime prompt tokens and attention. The model is reported at 1.3M parameters and 336M MACs in floating point, and at 1.22 MB after INT8 post-training quantization. On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively, while the quantized deployment runs at 14.3 ms on the IMX500 at approximately 86 MACs per cycle (Bonazzi et al., 23 Jun 2025).

1. Problem setting and deployment target

PicoSAM2 is positioned for real-time, on-device segmentation in latency-sensitive and privacy-aware settings such as smart glasses and IoT devices. Its design target is not generic edge inference in the abstract, but in-sensor execution under the operator, memory, and input restrictions of the Sony IMX500. The relevant constraints reported for that sensor are a total model size below 8 MB, RGB-only input, and a limited ONNX operator set; PicoSAM2 is therefore constructed to fit memory, avoid unsupported operators, and remove runtime prompt-processing overheads (Bonazzi et al., 23 Jun 2025).

The deployment model is explicitly on-camera. Sensor-captured RGB frames are processed directly by the IMX500 DSP, and masks are transmitted via SPI. This arrangement reduces bandwidth demands and keeps raw frames on device, which the paper links to privacy-preserving vision without cloud or host processing (Bonazzi et al., 23 Jun 2025).

A central design choice is that prompting is retained, but only in a restricted form compatible with the hardware. Instead of explicit point, box, or mask branches, PicoSAM2 uses an implicit single-point prompt realized through image cropping: a user-selected point is re-centered before inference, so the network receives only RGB input while still acting as a promptable segmenter. The paper refers to this as “fixed-point prompt encoding” (Bonazzi et al., 23 Jun 2025).

2. Network architecture and prompt formulation

The model topology is a U-Net-style encoder-decoder built from depthwise separable convolutions, up/down sampling, and skip connections. The encoder and decoder use quantization-friendly separable convolution blocks consisting of depthwise $3 \times 3$ followed by $1 \times 1$ pointwise convolutions, standard activations such as ReLU, and pyramid features. Transformer blocks are avoided, both to minimize cost and to remain within the IMX500-supported ONNX operator subset (Bonazzi et al., 23 Jun 2025).

This architecture is dense and CNN-based rather than token-based. At runtime, PicoSAM2 does not use prompt embeddings, prompt tokens, multi-branch prompt encoders, or attention-based prompt fusion. Prompt information is encoded implicitly by centering the crop around the selected point, so the network’s receptive field over the centered crop becomes the effective prompt interface. The execution consequence is zero prompt-overhead in the inference graph and no extra prompt buffers in memory (Bonazzi et al., 23 Jun 2025).

The teacher-student asymmetry is fundamental. During training, the teacher SAM2 uses explicit prompts, whereas the student receives only RGB crops centered at the prompt point. This transfers prompt-conditioned behavior into a model whose inference path is compatible with RGB-only sensing hardware. The paper does not disclose exact per-stage channel counts, kernel sizes per layer, or normalization choices, but it does identify the core construction as a separable-convolutional UNet with skip connections (Bonazzi et al., 23 Jun 2025).

3. Distillation objective and training formulation

The reported training regime uses COCO with AdamW and evaluates on LVIS. Resolution, augmentations, batch size, and training schedule details are not disclosed. The optimization objective combines a distillation term from SAM2 with a hard segmentation term, weighted by a confidence-driven factor $\lambda$ (Bonazzi et al., 23 Jun 2025).

The total loss is given as

$$L_{\text{total}} = \lambda L_{\text{MSE}} + (1 - \lambda)(0.5 L_{\text{BCE}} + 0.5 L_{\text{Dice}}).$$

Here, $L_{\text{MSE}}$ matches student logits to teacher logits, while the segmentation term combines balanced binary cross-entropy and Dice loss:

$$L_{\text{BCE}} = -\frac{1}{N} \sum_i \left[g_i \log \sigma(p_i) + (1-g_i)\log(1-\sigma(p_i))\right],$$

and

$$L_{\text{Dice}} = 1 - \frac{2 \sum_i p_i g_i + \epsilon}{\sum_i p_i + \sum_i g_i + \epsilon}.$$

The paper further states that there is no separate prompt loss term. In the decomposition $$L = \alpha L_{\text{seg}} + \beta L_{\text{distill}} + \gamma L_{\text{prompt}}$$, it sets $\gamma = 0$, with $\beta = \lambda$ and $1 \times 1$0 (Bonazzi et al., 23 Jun 2025).

Evaluation uses standard overlap metrics. The definitions reported are

$1 \times 1$1

$1 \times 1$2

and

$1 \times 1$3

Within this formulation, distillation is reported to improve LVIS performance by $1 \times 1$4 mIoU and $1 \times 1$5 mAP relative to purely supervised training, which the paper interprets as better generalization and precision (Bonazzi et al., 23 Jun 2025).

4. Quantization and in-sensor execution on the Sony IMX500

PicoSAM2 is quantized with static post-training INT8 quantization across weights and activations. The quantization mapping is written as

$1 \times 1$6

with dequantization

$1 \times 1$7

when a zero-point $1 \times 1$8 is used. The paper does not report per-layer bit-width variations; INT8 is used uniformly for the CNN layers (Bonazzi et al., 23 Jun 2025).

The quantized model, denoted Q-PicoSAM2, occupies 1.22 MB and performs 324M MACs. The slight reduction from 336M MACs is attributed to fusions or reordering. On the IMX500, the measured latency is 14.3 ms per image. Given 324M MACs and a 262.5 MHz DSP with 2304 MAC units, the effective throughput is reported as approximately 22.7 G MAC/s, or about 86 MACs/cycle; a later comparative review reports the same deployment at 86.2 MAC/cycle (Bonazzi et al., 23 Jun 2025, Capogrosso et al., 18 Feb 2026).

The hardware target is a stacked BSI-CMOS image sensor with integrated DSP optimized for CNN inference. The paper reports 2304 MAC units, a 262.5 MHz clock, 4.97 TOPS/W, RGB-only input, strict memory constraints, and ONNX operator restrictions. PicoSAM2 runs fully on-sensor as a CNN graph; prompt handling is offloaded to preprocessing, where the selected point is centered in a crop before inference. No transformer layers, attention layers, or custom operators are required (Bonazzi et al., 23 Jun 2025).

A concise summary of the reported operating points is as follows.

Variant	Accuracy and cost	Runtime
PicoSAM2 (FP32)	1.3M parameters; 336M MACs; 4.87 MB; 51.9% mIoU on COCO; 44.9% mIoU on LVIS	2.54 ms on NVIDIA T4
Q-PicoSAM2 (INT8)	324M MACs; 1.22 MB; 50.5% mIoU on COCO; 45.1% mIoU on LVIS	14.3 ms on IMX500

These numbers define the model’s core trade-off: the inference graph is deliberately simplified to satisfy in-sensor operator and memory limits while preserving promptability in a single-point form (Bonazzi et al., 23 Jun 2025).

5. Empirical results and comparative position among SAM-derived lightweight models

On COCO and LVIS, PicoSAM2 reports 51.9% and 44.9% mIoU in FP32. The quantized model reports 50.5% mIoU on COCO and 45.1% mIoU on LVIS. The same study states that PicoSAM2 is the only model among recent promptable SAM variants to meet the IMX500’s in-sensor compute and memory constraints (Bonazzi et al., 23 Jun 2025).

The comparison set reported in the paper places PicoSAM2 against SAM-H, FastSAM, TinySAM, EdgeSAM, and LiteSAM. The stated figures are: SAM-H at 2.97T MACs, 2.39 s, 635M parameters, and 2,420 MB; FastSAM at 443G MACs, 153.6 ms, 72.2M parameters, and 275.6 MB; TinySAM at 42G MACs, 38.4 ms, 9.7M parameters, and 37 MB; EdgeSAM at 42G MACs, 38.4 ms, 9.7M parameters, and 37 MB; and LiteSAM at 4.2G MACs, 16.0 ms, 4.2M parameters, and 16 MB. The paper attributes non-deployability on IMX500 to different causes across these baselines, including unsupported transformer components, prompt overhead, or excessive size (Bonazzi et al., 23 Jun 2025).

This comparison is not merely about model compression. The distinguishing axis is hardware compatibility under an RGB-only, operator-limited, memory-limited in-sensor runtime. In that sense, PicoSAM2 occupies a narrower but more deployment-specific design point than general lightweight SAM derivatives. This suggests that its principal contribution is not maximal segmentation quality in isolation, but promptable segmentation under a sensor-resident execution model (Bonazzi et al., 23 Jun 2025).

A subsequent comparative review uses PicoSAM2 as a 336 million MAC segmentation workload on a $1 \times 1$9 RGB input with batch size 1 to stress compute-memory hierarchies across three low-power processors. For this workload, the reported results are 14.3 ms, 69.9 FPS, 86.2 MAC/cycle, 1359.6 MMAC/J, and 3.4 mJ·s EDP on IMX500; 13.7 ms, 73.0 FPS, 29.5 MAC/cycle, 21.5 MAC/J, and 206.8 mJ·s EDP on STM32N6; and 42.1 ms, 23.8 FPS, 20.8 MAC/cycle, 182.15 MMAC/J, and 74.88 mJ·s EDP on GAP9. The review interprets IMX500 as having the highest utilization and lowest energy-delay product, STM32N6 as having the lowest raw latency, and GAP9 as offering the best energy efficiency within microcontroller-class power budgets (Capogrosso et al., 18 Feb 2026).

6. Scope, limitations, and relation to adjacent work

The paper explicitly identifies several limitations. PicoSAM2 remains below full SAM2 in accuracy because of capacity and architectural simplifications. Its runtime prompting is restricted to single-point prompts; boxes and masks are not supported at inference. Complex scenes with multiple overlapping objects may require multiple crops or external orchestration. The current pipeline targets single-object masks per prompt and static images, and future extensions are framed in terms of multi-prompt orchestration, temporal priors, hardware-aware architecture search, pruning, and richer distillation strategies (Bonazzi et al., 23 Jun 2025).

The name should also be distinguished from other “SAM2”-related or “Pico”-adjacent systems. “TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model” is a lightweight video segmentation model built around SAM 2 memory-bank compression, memory quality management, joint spatial-temporal token compression, and a RepViT image encoder; it addresses video segmentation efficiency rather than in-sensor promptable image segmentation (Ding et al., 18 May 2026). Conversely, “SPIDER, a Waveform Digitizer ASIC for picosecond timing in LHCb PicoCal” is an ASIC for timing measurement in high-energy physics, and its paper explicitly states that it does not mention or describe a device called PicoSAM2 (Alvado et al., 19 Dec 2025).

These distinctions matter because they separate three different problem formulations: memory-compressed SAM 2 video segmentation, promptable in-sensor image segmentation, and picosecond waveform digitization. PicoSAM2 belongs to the second category. Its defining features are the fixed spatial prompt prior, dense CNN implementation, static INT8 deployment, and demonstrated operation on the Sony IMX500 under strict memory and operator constraints (Bonazzi et al., 23 Jun 2025).