Intel Neural Compute Stick (INCS)

• Intel Neural Compute Stick (INCS) is a family of USB-attached neural inference accelerators built on Movidius VPUs for embedded and edge AI applications.
• The device integrates dedicated neural compute engines and programmable SHAVE processors, supporting heterogeneous execution for both CNN and classical computer vision tasks.
• INCS is optimized with software stacks like OpenVINO and hardware-aware model design techniques, ensuring reliable low-latency performance and efficient power usage.

Searching arXiv for recent and foundational papers on Intel Neural Compute Stick / Movidius VPU. Searching "Intel Neural Compute Stick Movidius Myriad X" Intel Neural Compute Stick (INCS) denotes Intel’s USB-attached neural inference accelerator built around the Movidius Vision Processing Unit (VPU). In the research literature, the term spans the original Neural Compute Stick, which encapsulates a Movidius Myriad 2 VPU (variant MA2450), and the Intel Neural Compute Stick 2 (NCS2), which is based on the Intel Movidius Myriad X VPU (Rivas-Gomez et al., 2018, Mohammadi et al., 2023). Across these variants, the device is treated as a host-connected co-processor for CNN inference and, in Myriad X deployments, as a heterogeneous edge-AI target in which a dedicated neural compute engine is coupled to programmable SHAVE vector processors and a scratch-pad-centric memory hierarchy (Leon et al., 2024).

1. Terminology, lineage, and device scope

The published record uses several closely related names. The original Neural Compute Stick is described as a USB-3.0 dongle encapsulating a Movidius Myriad 2 VPU, while the NCS2 is a USB-based accelerator built on the Myriad X VPU (Rivas-Gomez et al., 2018, Mohammadi et al., 2023). A later Myriad X technical overview explicitly characterizes INCS as “Intel’s Myriad X VPU in a USB form-factor” (Leon et al., 2024). This suggests that “INCS” is best understood as a family label used in comparative edge-inference work rather than as a single immutable hardware revision.

Platform in the literature	Underlying VPU	Reported characteristics
Neural Compute Stick	Movidius Myriad 2 (MA2450)	USB-3.0 dongle; 12 SHAVEs at 600 MHz; 2 MB CMX; SIPP; die dissipates $< 1$ W; stick draws under 2.5 W
Neural Compute Stick 2 / INCS	Intel Movidius Myriad X	16 programmable SHAVE cores; Neural Compute Engine; base clock 700 MHz; FP16 support; Myriad X / INCS overview reports 512 MB LPDDR4 and 2.5 MB CMX

Published descriptions are not uniform on the external host link. The Myriad X / INCS overview lists USB 2.0 as the interface used by INCS to a host PC or Raspberry Pi, whereas the Myriad 2 and NCS2 descriptions specify USB 3.0 (Leon et al., 2024, Rivas-Gomez et al., 2018, Mohammadi et al., 2023). That discrepancy is part of the literature as reported and should be read as a difference between source descriptions rather than a settled architectural statement.

2. Microarchitecture and execution resources

The Myriad family is organized around SHAVE vector processors, on-chip scratchpad storage, lightweight control cores, and fixed-function or semi-fixed-function acceleration blocks. In the Myriad 2-based stick, two little RISC cores handle USB traffic and scheduling, load a compiled CNN “graph” into the die, and communicate with the host via the Neural Compute API. The compute substrate comprises 12 identical SHAVEs, each clocked at 600 MHz and controlled via Variable-Length VLIW packets. Every SHAVE includes a 128-bit Vector Arithmetic Unit, a 128-bit Compare-and-Move Unit, a 32-bit Scalar Arithmetic Unit, a 32-bit Integer Arithmetic Unit, two 64-bit Load/Store Units, and native support for 8/16/32-bit integer operations, FP16, and FP32 (Rivas-Gomez et al., 2018).

All 12 Myriad 2 SHAVEs share a low-latency, multi-ported 2 MB scratchpad called the Connection Matrix (CMX), subdivided into 16 independently arbitrated 128 KB banks. A second subsystem, the Streaming Image Processing Pipeline (SIPP), provides fixed-function $5 \times 5$ hardware kernels such as tone-mapping, HoG filters, and Harris corner detection. Off-chip global memory is served by a stacked 4 GB LPDDR3 interface. The paper reports a theoretical FP16 rate of approximately $921.6$ GFLOPS and notes that the chip’s peak FP16 rate is advertised at 1 TFLOPS (Rivas-Gomez et al., 2018).

The Myriad X generation adds a dedicated Neural Compute Engine (NCE) for convolutional and fully-connected layers. In the Myriad X / INCS overview, the NCE supports FP16 convolutions, fused convolutions plus bias plus activation, and matrix-multiply kernels. The general-purpose side consists of 16 SHAVE processors described as 128-bit VLIW vector cores running up to 700 MHz. The same overview reports a memory hierarchy with 512 MB LPDDR4 DRAM, a 2.5 MB on-chip scratch-pad partitioned into 16 CMX slices, one per SHAVE, and caches for two LEON4 RISC cores and for SHAVEs (Leon et al., 2024).

The execution model exposed by these descriptions is heterogeneous rather than monolithic. Convolutions and fully-connected layers are preferentially mapped to the NCE on Myriad X, whereas custom CV kernels, pre-processing, post-processing, small arithmetic kernels, permutations, and memory re-shapes are assigned to SHAVEs (Leon et al., 2024). In this architecture, throughput and efficiency are commonly expressed with compact formulas such as

$$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$

$S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$

and, in comparative benchmarking,

$$\mathrm{Throughput}_W = \frac{\text{images/s}}{\mathrm{TDP\,(W)}}.$$

These metrics recur across the literature as the primary abstractions for synchronous throughput, parallel speedup, and energy efficiency (Leon et al., 2024, Rivas-Gomez et al., 2018).

3. Software toolchains and deployment flows

The software stack has evolved from the original NCSDK workflow to OpenVINO-centric deployment. For the Myriad 2 stick, the host toolchain uses a Caffe-based front-end for model definition and training, a model “freezer” that converts a pre-trained GoogLeNet .caffemodel into a Movidius “graph” (.mvnc), and the Intel Neural Compute SDK, which exposes NCAPI in C/C++. The key runtime calls are mvncOpenDevice(), mvncAllocateGraph(), mvncLoadTensor(), mvncGetResult(), mvncUnloadGraph(), and mvncCloseDevice(). Importantly, mvncLoadTensor() is non-blocking and mvncGetResult() is blocking, so host-side decoding or pre/post-processing can overlap with VPU execution (Rivas-Gomez et al., 2018).

In Myriad X and NCS2 workflows, OpenVINO is the dominant toolchain. One reported path begins with a TensorFlow frozen graph, runs the Model Optimizer to generate IR (.xml and .bin), partitions the IR graph into NCE subgraphs and SHAVE kernels through the Myriad X compiler, and generates a .blob file loaded via the mvNCInferRequest API (Leon et al., 2024). Another reported path exports a TensorFlow SavedModel to OpenVINO IR in FP16 using OpenVINO 2020.4, after which the Inference Engine Python3/C++ API loads the IR onto the MYRIAD device, manages device memory, and schedules inference (Mohammadi et al., 2023). A benchmarking study aimed at broad model coverage instead instantiates a Keras model in TensorFlow, exports to ONNX via tf2onnx, and then applies the OpenVINO Model Optimizer and compiler with the “latency hint” before single-stream timing (Tobiasz et al., 2023).

Across these flows, graph-level optimization is central. Reported transformations include constant-folding, layer fusion, shape inference, precision reduction from FP32 to FP16, and fusion of conv + ReLU and conv + batch-norm where applicable (Mohammadi et al., 2023). For the original NCS, input conversion from host-side FP32 to FP16 uses the OpenEXR half-precision class, and multi-stick saturation is achieved with OpenMP threads, one per stick, under a static round-robin scheduler (Rivas-Gomez et al., 2018). These workflows establish INCS as a compiled-inference target rather than a device programmed directly at the graph-construction level.

4. Workload mapping, hybrid AI/CV execution, and hardware-aware design

A detailed Myriad X case study is the satellite pose-estimation system built around a ResNet-50-based UrsoNet network plus a classical CV tracker. The UrsoNet deployment removes the global average-pool and final fully connected layer of ResNet-50, inserts a $3 \times 3$ convolution bottleneck with stride 2 and width 128, and uses two FP16 fully connected heads of width 512 to compute satellite location $(x,y,z)$ and orientation, for example a quaternion. The network accepts $512 \times 512 \times 3$ RGB input resampled from $1024 \times 1024$ and is trained on the “soyuz_hard” dataset (Leon et al., 2024).

The DNN serves as a “Lost-In-Space” initializer that outputs an initial 6-DOF pose. A classical CV pipeline then refines and tracks that pose through grayscale conversion of the $5 \times 5$0 RGB frame, Canny edge detection with five stages, depth-map rendering of the 3D satellite mesh under the current pose, perpendicular edge matching between image edges and depth edges, and robust pose refinement via RANSAC and linear algebra (Leon et al., 2024). The resulting dataflow is explicitly hybrid: coarse pose from the DNN feeds depth rendering, refined pose is produced by the tracker, and that refined pose seeds the next frame.

This hybrid pipeline also illustrates the Myriad X mapping policy. Convolutions and fully connected layers are sent to the NCE, BatchNorm is fused into convolution bias and handled as Add on SHAVEs, and small element-wise operations, transposes, activations, and memory re-shapes are executed on SHAVEs. Weights and large feature maps reside in DRAM and are double-buffered via DMA into CMX slices, while intermediate tensors smaller than 64 KB are placed directly in CMX for on-the-fly reuse (Leon et al., 2024). The low-level optimization repertoire includes stripe tiling across horizontal image bands, DMA prefetch from DRAM to CMX, double-linked row buffers that maintain a sliding window in CMX, packing of u8/u16 into 128-bit vectors, intra-instruction $5 \times 5$1 FP16 MACs, and synchronized barriers across SHAVEs (Leon et al., 2024).

At the model-design level, hardware-aware NAS has been extended to the Movidius VPU family through two latency-estimation strategies: direct pre-collected hardware cost on device and the device-specific hardware-cost model VPUNN, an open-sourced 2-layer MLP that predicts cycle count from operator metadata (Xu et al., 2023). The NAS objective augments ProxylessNAS with an expected-latency regularizer,

$5 \times 5$2

and

$5 \times 5$3

so that architecture search internalizes VPU cost rather than treating latency as an after-the-fact constraint (Xu et al., 2023). In reported results, this produces classification networks with $5 \times 5$4 FPS acceleration over Mobilenet-v2-1.4 and $5 \times 5$5 acceleration over ResNet-50 at the same accuracy score, and a super-resolution network with $5 \times 5$6 PSNR and $5 \times 5$7 higher FPS than EDSR3 (Xu et al., 2023).

5. Empirical performance and efficiency

Measured performance varies substantially with generation, workload, and evaluation protocol. On the original Myriad 2 NCS, GoogLeNet inference over ILSVRC 2012 validation yields 9.93 img/s at 100.7 ms latency for a single stick and 77.2 img/s at 12.9 ms latency for an 8-stick configuration. Energy efficiency remains near 3.9 img/$5 \times 5$8 up to 8 sticks, and the 8-stick system reaches Top-1 accuracy of 31.92% versus 32.01% for CPU FP32. Reported normalized speedups from 1 to 8 sticks are $5 \times 5$9, which the paper describes as nearly linear scaling (Rivas-Gomez et al., 2018).

On Myriad X, the satellite pose-estimation system reports overall synchronous AI latency of 374–392 ms at 1 MP input, corresponding to 2.6–2.7 FPS, with NCE inference at $921.6$0 taking 373 ms. The corresponding CV tracking pipeline at 1 MP grayscale reports 218–341 ms total sequential latency and up to 5.1 FPS with streaming. Power is reported as core $921.6$1 W, DRAM $921.6$2 W, total $921.6$3 W, with 10% saved by power-island shutdown. Pre-processing is reported as up to 1000 FPS for bilinear resampling, with bilinear at 1 ms, bicubic at 6 ms, and Lanczos at 19 ms (Leon et al., 2024).

A facial-expression-recognition deployment on NCS2 reports 97.46% accuracy on CK+, mean latency 2.35 ms, dynamic power 2.08 W, throughput $921.6$4 ms $921.6$5 inf/s, and energy per inference

$921.6$6

In the same study, NCS2 is approximately $921.6$7 faster than a Raspberry Pi CPU baseline but approximately $921.6$8 slower than a Coral USB TPU, while reducing per-inference energy by approximately 27% versus the Raspberry Pi CPU (Mohammadi et al., 2023).

Workload	Reported INCS result	Additional context
FER best selected model	2.35 ms; $921.6$9 inf/s; 97.46% accuracy	Dynamic power 2.08 W; energy 4.88 mJ
MobileNetV2 ($$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$0, no head, $$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$1)	$$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$2 ms; 40.31 FPS	Single-stream OpenVINO timing
ResNet50 (no head, $$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$3)	$$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$4 ms; 26.03 FPS	Single-stream OpenVINO timing
InceptionV3 (no head, $$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$5)	$$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$6 ms; 34.76 FPS	Single-stream OpenVINO timing
VGG16 (no head, $$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$7)	$$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$8 ms; 1.87 FPS	Large-resolution feature extractor

A broad single-stream benchmark further shows that INCS latency rises predictably with model scale and input area: EfficientNetV2B0 at $$\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},$$9 is $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$0 ms, ResNet50 at $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$1 is $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$2 ms, and InceptionV3 at $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$3 is $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$4 ms. Adding a standard 1000-class ImageNet head or a 5-class head typically adds 1–3 ms on top of the no-head feature-extractor baseline (Tobiasz et al., 2023). In a separate satellite benchmark, UrsoNet at $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$5 on INCS plus Raspberry Pi 3 is reported at 1.7 FPS, corresponding to 0.34 FPS/W, versus 0.4 FPS on Jetson Nano ARM A57 at 10 W and 1.3 FPS on Jetson Nano GPU at 10 W (Leon et al., 2024).

6. Comparative position, limitations, and sustainability

The most consistent comparative theme is a trade-off between universality, deterministic latency, and peak speed. One edge-platform study characterizes INCS as the “most universal” accelerator because it ran every tested model and every tested resolution up to 1024, while Coral platforms could fail or compile-error beyond roughly 600 px on large networks and Jetson Nano could run out of memory. The same study also reports that INCS is 8–10× slower than a Coral USB on small models such as MobileNetV2 at $S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},$6, and 2–3× slower than Jetson Nano in FP16 mode on many mid-size networks, but with very low standard deviation, approximately 0.3–0.7 ms, compared with much higher Jetson Nano jitter (Tobiasz et al., 2023).

Energy-efficiency claims are likewise workload-sensitive. Early VPU work on the Myriad 2 stick emphasizes a die power envelope of less than 1 W and a stick draw under 2.5 W even at peak data transfer, while the Myriad X satellite system reports a total of approximately 2.0 W during hybrid AI/CV pose estimation (Rivas-Gomez et al., 2018, Leon et al., 2024). By contrast, a later sustainability study reports substantially larger active power values for heavier models on INCS: ResNet-50 at 12.65 W and 20.36 J per inference, MobileSSD at 18.18 W and 25.82 J, TinyBERT at 43.20 W and 301.10 J, and Phi-2 at 55.50 W and 807.48 J (Sobhani et al., 30 Jul 2025). This suggests that blanket descriptions of INCS as uniformly low-power are only valid relative to specific workloads, host configurations, and measurement protocols.

The same sustainability study broadens the applicability boundary of the platform by showing that INCS can execute traditional ML, neural networks, deep learning models, and even compact LLMs, but with sharply different operating points. Decision Tree is reported at 0.14 s and 0.36 J per inference, ANN at 0.38 s and 0.81 J, CNN at 0.79 s and 3.07 J, ResNet-50 at 1.61 s and 20.36 J, and TinyBERT at 6.97 s and 301.10 J (Sobhani et al., 30 Jul 2025). A plausible implication is that INCS is well aligned with small and medium-sized CNNs and feature-extractor workloads, but becomes marginal for heavier detectors and especially for transformer-scale NLP unless aggressive tuning is applied.

Published guidance on precision and quantization is not fully uniform. One benchmark states that INCS only accepts FP16/FP32 ONNX to OpenVINO IR and reports no INT8 quantization path (Tobiasz et al., 2023). A later sustainability study, however, describes OpenVINO IR on INCS with graph-level optimizations including operator fusion, pruning, and quantization, and recommends INT8 quantization and operator fusion for improved performance (Sobhani et al., 30 Jul 2025). Similar device-specific guidance converges on several operational practices despite that discrepancy: exploit FP16 when targeting Myriad X, warm up before timing, separate enqueue from dequeue when using NCAPI, overlap host-side work with device execution, match model structure to the VPU’s channel-block and scratch-pad behavior, and avoid architectures whose latency or memory behavior is incompatible with the device envelope (Rivas-Gomez et al., 2018, Mohammadi et al., 2023, Leon et al., 2024, Xu et al., 2023).

In aggregate, the literature places INCS at the intersection of embedded heterogeneous computing, low-power CNN inference, and hardware-aware model design. Its distinguishing properties are not absolute throughput leadership, but a combination of VPU-centric programmability, stable single-stream latency, support for a wide range of model families, and a software path that integrates classical CV, CNN inference, and hardware-aware NAS into a single edge deployment target (Tobiasz et al., 2023, Leon et al., 2024, Xu et al., 2023).