---
title: 'PicoSAM2: Lightweight In-Sensor Segmentation'
url: https://www.emergentmind.com/topics/picosam2
type: topic
---

# PicoSAM2: Lightweight In-Sensor Segmentation

Searching arXiv for PicoSAM2 and closely related papers to ground the article in current literature.
PicoSAM2 is a lightweight, promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500. It is described as a depthwise separable U-Net distilled from Segment Anything Model 2 (SAM2), with fixed-point prompt encoding that eliminates runtime prompt tokens and attention. The model is reported at 1.3M parameters and 336M MACs in floating point, and at 1.22 MB after INT8 post-training quantization. On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively, while the quantized deployment runs at 14.3 ms on the IMX500 at approximately 86 MACs per cycle [2506.18807].

## 1. Problem setting and deployment target

PicoSAM2 is positioned for real-time, on-device segmentation in latency-sensitive and privacy-aware settings such as smart glasses and IoT devices. Its design target is not generic edge inference in the abstract, but in-sensor execution under the operator, memory, and input restrictions of the Sony IMX500. The relevant constraints reported for that sensor are a total model size below 8 MB, RGB-only input, and a limited ONNX operator set; PicoSAM2 is therefore constructed to fit memory, avoid unsupported operators, and remove runtime prompt-processing overheads [2506.18807].

The deployment model is explicitly on-camera. Sensor-captured RGB frames are processed directly by the IMX500 DSP, and masks are transmitted via SPI. This arrangement reduces bandwidth demands and keeps raw frames on device, which the paper links to privacy-preserving vision without cloud or host processing [2506.18807].

A central design choice is that prompting is retained, but only in a restricted form compatible with the hardware. Instead of explicit point, box, or mask branches, PicoSAM2 uses an implicit single-point prompt realized through image cropping: a user-selected point is re-centered before inference, so the network receives only RGB input while still acting as a promptable segmenter. The paper refers to this as “fixed-point prompt encoding” [2506.18807].

## 2. Network architecture and prompt formulation

The model topology is a U-Net-style encoder-decoder built from depthwise separable convolutions, up/down sampling, and skip connections. The encoder and decoder use quantization-friendly separable convolution blocks consisting of depthwise $3 \times 3$ followed by $1 \times 1$ pointwise convolutions, standard activations such as ReLU, and pyramid features. Transformer blocks are avoided, both to minimize cost and to remain within the IMX500-supported ONNX operator subset [2506.18807].

This architecture is dense and CNN-based rather than token-based. At runtime, PicoSAM2 does not use prompt embeddings, prompt tokens, multi-branch prompt encoders, or attention-based prompt fusion. Prompt information is encoded implicitly by centering the crop around the selected point, so the network’s receptive field over the centered crop becomes the effective prompt interface. The execution consequence is zero prompt-overhead in the inference graph and no extra prompt buffers in memory [2506.18807].

The teacher-student asymmetry is fundamental. During training, the teacher SAM2 uses explicit prompts, whereas the student receives only RGB crops centered at the prompt point. This transfers prompt-conditioned behavior into a model whose inference path is compatible with RGB-only sensing hardware. The paper does not disclose exact per-stage channel counts, kernel sizes per layer, or normalization choices, but it does identify the core construction as a separable-convolutional UNet with skip connections [2506.18807].

## 3. Distillation objective and training formulation

The reported training regime uses COCO with AdamW and evaluates on LVIS. Resolution, augmentations, batch size, and training schedule details are not disclosed. The optimization objective combines a distillation term from SAM2 with a hard segmentation term, weighted by a confidence-driven factor $\lambda$ [2506.18807].

The total loss is given as
$$
L_{\text{total}} = \lambda L_{\text{MSE}} + (1 - \lambda)(0.5 L_{\text{BCE}} + 0.5 L_{\text{Dice}}).
$$

Here, $L_{\text{MSE}}$ matches student logits to teacher logits, while the segmentation term combines balanced binary cross-entropy and Dice loss:
$$
L_{\text{BCE}} = -\frac{1}{N} \sum_i \left[g_i \log \sigma(p_i) + (1-g_i)\log(1-\sigma(p_i))\right],
$$
and
$$
L_{\text{Dice}} = 1 - \frac{2 \sum_i p_i g_i + \epsilon}{\sum_i p_i + \sum_i g_i + \epsilon}.
$$

The paper further states that there is no separate prompt loss term. In the decomposition $L = \alpha L_{\text{seg}} + \beta L_{\text{distill}} + \gamma L_{\text{prompt}}$, it sets $\gamma = 0$, with $\beta = \lambda$ and $\alpha = 1 - \lambda$ [2506.18807].

Evaluation uses standard overlap metrics. The definitions reported are
$$
\mathrm{IoU} = \frac{|P \cap G|}{|P \cup G|},
$$
$$
\mathrm{mIoU} = \frac{1}{K}\sum_{k=1}^{K}\mathrm{IoU}_k,
$$
and
$$
\mathrm{mAP} = \frac{1}{T}\sum_t AP(t).
$$
Within this formulation, distillation is reported to improve LVIS performance by $+3.5\%$ mIoU and $+5.1\%$ mAP relative to purely supervised training, which the paper interprets as better generalization and precision [2506.18807].

## 4. Quantization and in-sensor execution on the Sony IMX500

PicoSAM2 is quantized with static post-training INT8 quantization across weights and activations. The quantization mapping is written as
$$
Q(x) = \operatorname{clip}(\operatorname{round}(x/s), q_{\min}, q_{\max}),
$$
with dequantization
$$
D(q) = s \cdot q + z
$$
when a zero-point $z$ is used. The paper does not report per-layer bit-width variations; INT8 is used uniformly for the CNN layers [2506.18807].

The quantized model, denoted Q-PicoSAM2, occupies 1.22 MB and performs 324M MACs. The slight reduction from 336M MACs is attributed to fusions or reordering. On the IMX500, the measured latency is 14.3 ms per image. Given 324M MACs and a 262.5 MHz DSP with 2304 MAC units, the effective throughput is reported as approximately 22.7 G MAC/s, or about 86 MACs/cycle; a later comparative review reports the same deployment at 86.2 MAC/cycle [2506.18807] [2603.08725].

The hardware target is a stacked BSI-CMOS image sensor with integrated DSP optimized for CNN inference. The paper reports 2304 MAC units, a 262.5 MHz clock, 4.97 TOPS/W, RGB-only input, strict memory constraints, and ONNX operator restrictions. PicoSAM2 runs fully on-sensor as a CNN graph; prompt handling is offloaded to preprocessing, where the selected point is centered in a crop before inference. No transformer layers, attention layers, or custom operators are required [2506.18807].

A concise summary of the reported operating points is as follows.

| Variant | Accuracy and cost | Runtime |
|---|---|---|
| PicoSAM2 (FP32) | 1.3M parameters; 336M MACs; 4.87 MB; 51.9% mIoU on COCO; 44.9% mIoU on LVIS | 2.54 ms on NVIDIA T4 |
| Q-PicoSAM2 (INT8) | 324M MACs; 1.22 MB; 50.5% mIoU on COCO; 45.1% mIoU on LVIS | 14.3 ms on IMX500 |

These numbers define the model’s core trade-off: the inference graph is deliberately simplified to satisfy in-sensor operator and memory limits while preserving promptability in a single-point form [2506.18807].

## 5. Empirical results and comparative position among SAM-derived lightweight models

On COCO and LVIS, PicoSAM2 reports 51.9% and 44.9% mIoU in FP32. The quantized model reports 50.5% mIoU on COCO and 45.1% mIoU on LVIS. The same study states that PicoSAM2 is the only model among recent promptable SAM variants to meet the IMX500’s in-sensor compute and memory constraints [2506.18807].

The comparison set reported in the paper places PicoSAM2 against SAM-H, FastSAM, TinySAM, EdgeSAM, and LiteSAM. The stated figures are: SAM-H at 2.97T MACs, 2.39 s, 635M parameters, and 2,420 MB; FastSAM at 443G MACs, 153.6 ms, 72.2M parameters, and 275.6 MB; TinySAM at 42G MACs, 38.4 ms, 9.7M parameters, and 37 MB; EdgeSAM at 42G MACs, 38.4 ms, 9.7M parameters, and 37 MB; and LiteSAM at 4.2G MACs, 16.0 ms, 4.2M parameters, and 16 MB. The paper attributes non-deployability on IMX500 to different causes across these baselines, including unsupported transformer components, prompt overhead, or excessive size [2506.18807].

This comparison is not merely about model compression. The distinguishing axis is hardware compatibility under an RGB-only, operator-limited, memory-limited in-sensor runtime. In that sense, PicoSAM2 occupies a narrower but more deployment-specific design point than general lightweight SAM derivatives. This suggests that its principal contribution is not maximal segmentation quality in isolation, but promptable segmentation under a sensor-resident execution model [2506.18807].

A subsequent comparative review uses PicoSAM2 as a 336 million MAC segmentation workload on a $3 \times 96 \times 96$ RGB input with batch size 1 to stress compute-memory hierarchies across three low-power processors. For this workload, the reported results are 14.3 ms, 69.9 FPS, 86.2 MAC/cycle, 1359.6 MMAC/J, and 3.4 mJ·s EDP on IMX500; 13.7 ms, 73.0 FPS, 29.5 MAC/cycle, 21.5 MAC/J, and 206.8 mJ·s EDP on STM32N6; and 42.1 ms, 23.8 FPS, 20.8 MAC/cycle, 182.15 MMAC/J, and 74.88 mJ·s EDP on GAP9. The review interprets IMX500 as having the highest utilization and lowest energy-delay product, STM32N6 as having the lowest raw latency, and GAP9 as offering the best energy efficiency within microcontroller-class power budgets [2603.08725].

## 6. Scope, limitations, and relation to adjacent work

The paper explicitly identifies several limitations. PicoSAM2 remains below full SAM2 in accuracy because of capacity and architectural simplifications. Its runtime prompting is restricted to single-point prompts; boxes and masks are not supported at inference. Complex scenes with multiple overlapping objects may require multiple crops or external orchestration. The current pipeline targets single-object masks per prompt and static images, and future extensions are framed in terms of multi-prompt orchestration, temporal priors, hardware-aware architecture search, pruning, and richer distillation strategies [2506.18807].

The name should also be distinguished from other “SAM2”-related or “Pico”-adjacent systems. “TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model” is a lightweight video segmentation model built around SAM 2 memory-bank compression, memory quality management, joint spatial-temporal token compression, and a RepViT image encoder; it addresses video segmentation efficiency rather than in-sensor promptable image segmentation [2605.18013]. Conversely, “SPIDER, a Waveform Digitizer ASIC for picosecond timing in LHCb PicoCal” is an ASIC for timing measurement in high-energy physics, and its paper explicitly states that it does not mention or describe a device called PicoSAM2 [2512.17355].

These distinctions matter because they separate three different problem formulations: memory-compressed SAM 2 video segmentation, promptable in-sensor image segmentation, and picosecond waveform digitization. PicoSAM2 belongs to the second category. Its defining features are the fixed spatial prompt prior, dense CNN implementation, static INT8 deployment, and demonstrated operation on the Sony IMX500 under strict memory and operator constraints [2506.18807].

Source: https://www.emergentmind.com/topics/picosam2