---
title: Intel Neural Compute Stick (INCS)
url: https://www.emergentmind.com/topics/intel-neural-compute-stick-incs
type: topic
---

# Intel Neural Compute Stick (INCS)

Searching arXiv for recent and foundational papers on Intel Neural Compute Stick / Movidius VPU.
Searching arXiv: "Intel Neural Compute Stick Movidius Myriad X"
Intel Neural Compute Stick (INCS) denotes Intel’s USB-attached neural inference accelerator built around the Movidius Vision Processing Unit (VPU). In the research literature, the term spans the original Neural Compute Stick, which encapsulates a Movidius Myriad 2 VPU (variant MA2450), and the Intel Neural Compute Stick 2 (NCS2), which is based on the Intel Movidius Myriad X VPU [1810.04150] [2305.15422]. Across these variants, the device is treated as a host-connected co-processor for CNN inference and, in Myriad X deployments, as a heterogeneous edge-AI target in which a dedicated neural compute engine is coupled to programmable SHAVE vector processors and a scratch-pad-centric memory hierarchy [2409.12939].

## 1. Terminology, lineage, and device scope

The published record uses several closely related names. The original Neural Compute Stick is described as a USB-3.0 dongle encapsulating a Movidius Myriad 2 VPU, while the NCS2 is a USB-based accelerator built on the Myriad X VPU [1810.04150] [2305.15422]. A later Myriad X technical overview explicitly characterizes INCS as “Intel’s Myriad X VPU in a USB form-factor” [2409.12939]. This suggests that “INCS” is best understood as a family label used in comparative edge-inference work rather than as a single immutable hardware revision.

| Platform in the literature | Underlying VPU | Reported characteristics |
|---|---|---|
| Neural Compute Stick | Movidius Myriad 2 (MA2450) | USB-3.0 dongle; 12 SHAVEs at 600 MHz; 2 MB CMX; SIPP; die dissipates \(< 1\) W; stick draws under 2.5 W |
| Neural Compute Stick 2 / INCS | Intel Movidius Myriad X | 16 programmable SHAVE cores; Neural Compute Engine; base clock 700 MHz; FP16 support; Myriad X / INCS overview reports 512 MB LPDDR4 and 2.5 MB CMX |

Published descriptions are not uniform on the external host link. The Myriad X / INCS overview lists USB 2.0 as the interface used by INCS to a host PC or Raspberry Pi, whereas the Myriad 2 and NCS2 descriptions specify USB 3.0 [2409.12939] [1810.04150] [2305.15422]. That discrepancy is part of the literature as reported and should be read as a difference between source descriptions rather than a settled architectural statement.

## 2. Microarchitecture and execution resources

The Myriad family is organized around SHAVE vector processors, on-chip scratchpad storage, lightweight control cores, and fixed-function or semi-fixed-function acceleration blocks. In the Myriad 2-based stick, two little RISC cores handle USB traffic and scheduling, load a compiled CNN “graph” into the die, and communicate with the host via the Neural Compute API. The compute substrate comprises 12 identical SHAVEs, each clocked at 600 MHz and controlled via Variable-Length VLIW packets. Every SHAVE includes a 128-bit Vector Arithmetic Unit, a 128-bit Compare-and-Move Unit, a 32-bit Scalar Arithmetic Unit, a 32-bit Integer Arithmetic Unit, two 64-bit Load/Store Units, and native support for 8/16/32-bit integer operations, FP16, and FP32 [1810.04150].

All 12 Myriad 2 SHAVEs share a low-latency, multi-ported 2 MB scratchpad called the Connection Matrix (CMX), subdivided into 16 independently arbitrated 128 KB banks. A second subsystem, the Streaming Image Processing Pipeline (SIPP), provides fixed-function \(5 \times 5\) hardware kernels such as tone-mapping, HoG filters, and Harris corner detection. Off-chip global memory is served by a stacked 4 GB LPDDR3 interface. The paper reports a theoretical FP16 rate of approximately \(921.6\) GFLOPS and notes that the chip’s peak FP16 rate is advertised at 1 TFLOPS [1810.04150].

The Myriad X generation adds a dedicated Neural Compute Engine (NCE) for convolutional and fully-connected layers. In the Myriad X / INCS overview, the NCE supports FP16 convolutions, fused convolutions plus bias plus activation, and matrix-multiply kernels. The general-purpose side consists of 16 SHAVE processors described as 128-bit VLIW vector cores running up to 700 MHz. The same overview reports a memory hierarchy with 512 MB LPDDR4 DRAM, a 2.5 MB on-chip scratch-pad partitioned into 16 CMX slices, one per SHAVE, and caches for two LEON4 RISC cores and for SHAVEs [2409.12939].

The execution model exposed by these descriptions is heterogeneous rather than monolithic. Convolutions and fully-connected layers are preferentially mapped to the NCE on Myriad X, whereas custom CV kernels, pre-processing, post-processing, small arithmetic kernels, permutations, and memory re-shapes are assigned to SHAVEs [2409.12939]. In this architecture, throughput and efficiency are commonly expressed with compact formulas such as
$$
\mathrm{FPS} = \frac{N_{\text{frames}}}{T_{\text{total}}},
$$
$$
S = \frac{T_{\text{serial}}}{T_{\text{parallel}}},
$$
and, in comparative benchmarking,
$$
\mathrm{Throughput}_W = \frac{\text{images/s}}{\mathrm{TDP\,(W)}}.
$$
These metrics recur across the literature as the primary abstractions for synchronous throughput, parallel speedup, and energy efficiency [2409.12939] [1810.04150].

## 3. Software toolchains and deployment flows

The software stack has evolved from the original NCSDK workflow to OpenVINO-centric deployment. For the Myriad 2 stick, the host toolchain uses a Caffe-based front-end for model definition and training, a model “freezer” that converts a pre-trained GoogLeNet `.caffemodel` into a Movidius “graph” (`.mvnc`), and the Intel Neural Compute SDK, which exposes NCAPI in C/C++. The key runtime calls are `mvncOpenDevice()`, `mvncAllocateGraph()`, `mvncLoadTensor()`, `mvncGetResult()`, `mvncUnloadGraph()`, and `mvncCloseDevice()`. Importantly, `mvncLoadTensor()` is non-blocking and `mvncGetResult()` is blocking, so host-side decoding or pre/post-processing can overlap with VPU execution [1810.04150].

In Myriad X and NCS2 workflows, OpenVINO is the dominant toolchain. One reported path begins with a TensorFlow frozen graph, runs the Model Optimizer to generate IR (`.xml` and `.bin`), partitions the IR graph into NCE subgraphs and SHAVE kernels through the Myriad X compiler, and generates a `.blob` file loaded via the `mvNCInferRequest` API [2409.12939]. Another reported path exports a TensorFlow SavedModel to OpenVINO IR in FP16 using OpenVINO 2020.4, after which the Inference Engine Python3/C++ API loads the IR onto the `MYRIAD` device, manages device memory, and schedules inference [2305.15422]. A benchmarking study aimed at broad model coverage instead instantiates a Keras model in TensorFlow, exports to ONNX via `tf2onnx`, and then applies the OpenVINO Model Optimizer and compiler with the “latency hint” before single-stream timing [2306.12093].

Across these flows, graph-level optimization is central. Reported transformations include constant-folding, layer fusion, shape inference, precision reduction from FP32 to FP16, and fusion of `conv + ReLU` and `conv + batch-norm` where applicable [2305.15422]. For the original NCS, input conversion from host-side FP32 to FP16 uses the OpenEXR half-precision class, and multi-stick saturation is achieved with OpenMP threads, one per stick, under a static round-robin scheduler [1810.04150]. These workflows establish INCS as a compiled-inference target rather than a device programmed directly at the graph-construction level.

## 4. Workload mapping, hybrid AI/CV execution, and hardware-aware design

A detailed Myriad X case study is the satellite pose-estimation system built around a ResNet-50-based UrsoNet network plus a classical CV tracker. The UrsoNet deployment removes the global average-pool and final fully connected layer of ResNet-50, inserts a \(3 \times 3\) convolution bottleneck with stride 2 and width 128, and uses two FP16 fully connected heads of width 512 to compute satellite location \((x,y,z)\) and orientation, for example a quaternion. The network accepts \(512 \times 512 \times 3\) RGB input resampled from \(1024 \times 1024\) and is trained on the “soyuz_hard” dataset [2409.12939].

The DNN serves as a “Lost-In-Space” initializer that outputs an initial 6-DOF pose. A classical CV pipeline then refines and tracks that pose through grayscale conversion of the \(1024 \times 1024\) RGB frame, Canny edge detection with five stages, depth-map rendering of the 3D satellite mesh under the current pose, perpendicular edge matching between image edges and depth edges, and robust pose refinement via RANSAC and linear algebra [2409.12939]. The resulting dataflow is explicitly hybrid: coarse pose from the DNN feeds depth rendering, refined pose is produced by the tracker, and that refined pose seeds the next frame.

This hybrid pipeline also illustrates the Myriad X mapping policy. Convolutions and fully connected layers are sent to the NCE, BatchNorm is fused into convolution bias and handled as `Add` on SHAVEs, and small element-wise operations, transposes, activations, and memory re-shapes are executed on SHAVEs. Weights and large feature maps reside in DRAM and are double-buffered via DMA into CMX slices, while intermediate tensors smaller than 64 KB are placed directly in CMX for on-the-fly reuse [2409.12939]. The low-level optimization repertoire includes stripe tiling across horizontal image bands, DMA prefetch from DRAM to CMX, double-linked row buffers that maintain a sliding window in CMX, packing of `u8/u16` into 128-bit vectors, intra-instruction \(4 \times\) FP16 MACs, and synchronized barriers across SHAVEs [2409.12939].

At the model-design level, hardware-aware NAS has been extended to the Movidius VPU family through two latency-estimation strategies: direct pre-collected hardware cost on device and the device-specific hardware-cost model VPUNN, an open-sourced 2-layer MLP that predicts cycle count from operator metadata [2305.03739]. The NAS objective augments ProxylessNAS with an expected-latency regularizer,
$$
E[\mathrm{latency}] = \sum_{i=1}^{N} \sum_{j=1}^{M} p_i^j \cdot F(o_i^j),
$$
and
$$
\mathrm{Loss} = \mathrm{Loss}_{CE}(\mathrm{network}) + \lambda_1 \cdot \|w\|_2^2 + \lambda_2 \cdot E[\mathrm{latency}],
$$
so that architecture search internalizes VPU cost rather than treating latency as an after-the-fact constraint [2305.03739]. In reported results, this produces classification networks with \(1.3\times\) FPS acceleration over Mobilenet-v2-1.4 and \(2.2\times\) acceleration over ResNet-50 at the same accuracy score, and a super-resolution network with \(1.08\times\) PSNR and \(6\times\) higher FPS than EDSR3 [2305.03739].

## 5. Empirical performance and efficiency

Measured performance varies substantially with generation, workload, and evaluation protocol. On the original Myriad 2 NCS, GoogLeNet inference over ILSVRC 2012 validation yields 9.93 img/s at 100.7 ms latency for a single stick and 77.2 img/s at 12.9 ms latency for an 8-stick configuration. Energy efficiency remains near 3.9 img/\((\mathrm{s}\cdot\mathrm{W})\) up to 8 sticks, and the 8-stick system reaches Top-1 accuracy of 31.92% versus 32.01% for CPU FP32. Reported normalized speedups from 1 to 8 sticks are \(1\times \rightarrow 1.98\times \rightarrow 3.91\times \rightarrow 7.78\times\), which the paper describes as nearly linear scaling [1810.04150].

On Myriad X, the satellite pose-estimation system reports overall synchronous AI latency of 374–392 ms at 1 MP input, corresponding to 2.6–2.7 FPS, with NCE inference at \(512 \times 512\) taking 373 ms. The corresponding CV tracking pipeline at 1 MP grayscale reports 218–341 ms total sequential latency and up to 5.1 FPS with streaming. Power is reported as core \(\approx 1.9\) W, DRAM \(\approx 0.1\) W, total \(\approx 2.0\) W, with 10% saved by power-island shutdown. Pre-processing is reported as up to 1000 FPS for bilinear resampling, with bilinear at 1 ms, bicubic at 6 ms, and Lanczos at 19 ms [2409.12939].

A facial-expression-recognition deployment on NCS2 reports 97.46% accuracy on CK+, mean latency 2.35 ms, dynamic power 2.08 W, throughput \(1/2.35\) ms \(\approx 425\) inf/s, and energy per inference
$$
E = P \times t \approx 2.08\,\mathrm{W} \times 2.35\,\mathrm{ms} = 4.88\,\mathrm{mJ}.
$$
In the same study, NCS2 is approximately \(2\times\) faster than a Raspberry Pi CPU baseline but approximately \(1.3\times\) slower than a Coral USB TPU, while reducing per-inference energy by approximately 27% versus the Raspberry Pi CPU [2305.15422].

| Workload | Reported INCS result | Additional context |
|---|---|---|
| FER best selected model | 2.35 ms; \(\approx 425\) inf/s; 97.46% accuracy | Dynamic power 2.08 W; energy 4.88 mJ |
| MobileNetV2 (\(\alpha=1\), no head, \(224 \times 224\)) | \(24.81 \pm 0.31\) ms; 40.31 FPS | Single-stream OpenVINO timing |
| ResNet50 (no head, \(224 \times 224\)) | \(38.42 \pm 0.27\) ms; 26.03 FPS | Single-stream OpenVINO timing |
| InceptionV3 (no head, \(224 \times 224\)) | \(28.77 \pm 0.28\) ms; 34.76 FPS | Single-stream OpenVINO timing |
| VGG16 (no head, \(512 \times 512\)) | \(533.86 \pm 0.71\) ms; 1.87 FPS | Large-resolution feature extractor |

A broad single-stream benchmark further shows that INCS latency rises predictably with model scale and input area: EfficientNetV2B0 at \(224 \times 224\) is \(49.50 \pm 0.38\) ms, ResNet50 at \(512 \times 512\) is \(171.78 \pm 0.66\) ms, and InceptionV3 at \(512 \times 512\) is \(176.08 \pm 0.62\) ms. Adding a standard 1000-class ImageNet head or a 5-class head typically adds 1–3 ms on top of the no-head feature-extractor baseline [2306.12093]. In a separate satellite benchmark, UrsoNet at \(512 \times 640\) on INCS plus Raspberry Pi 3 is reported at 1.7 FPS, corresponding to 0.34 FPS/W, versus 0.4 FPS on Jetson Nano ARM A57 at 10 W and 1.3 FPS on Jetson Nano GPU at 10 W [2409.12939].

## 6. Comparative position, limitations, and sustainability

The most consistent comparative theme is a trade-off between universality, deterministic latency, and peak speed. One edge-platform study characterizes INCS as the “most universal” accelerator because it ran every tested model and every tested resolution up to 1024, while Coral platforms could fail or compile-error beyond roughly 600 px on large networks and Jetson Nano could run out of memory. The same study also reports that INCS is 8–10× slower than a Coral USB on small models such as MobileNetV2 at \(224 \times 224\), and 2–3× slower than Jetson Nano in FP16 mode on many mid-size networks, but with very low standard deviation, approximately 0.3–0.7 ms, compared with much higher Jetson Nano jitter [2306.12093].

Energy-efficiency claims are likewise workload-sensitive. Early VPU work on the Myriad 2 stick emphasizes a die power envelope of less than 1 W and a stick draw under 2.5 W even at peak data transfer, while the Myriad X satellite system reports a total of approximately 2.0 W during hybrid AI/CV pose estimation [1810.04150] [2409.12939]. By contrast, a later sustainability study reports substantially larger active power values for heavier models on INCS: ResNet-50 at 12.65 W and 20.36 J per inference, MobileSSD at 18.18 W and 25.82 J, TinyBERT at 43.20 W and 301.10 J, and Phi-2 at 55.50 W and 807.48 J [2507.23093]. This suggests that blanket descriptions of INCS as uniformly low-power are only valid relative to specific workloads, host configurations, and measurement protocols.

The same sustainability study broadens the applicability boundary of the platform by showing that INCS can execute traditional ML, neural networks, deep learning models, and even compact language models, but with sharply different operating points. Decision Tree is reported at 0.14 s and 0.36 J per inference, ANN at 0.38 s and 0.81 J, CNN at 0.79 s and 3.07 J, ResNet-50 at 1.61 s and 20.36 J, and TinyBERT at 6.97 s and 301.10 J [2507.23093]. A plausible implication is that INCS is well aligned with small and medium-sized CNNs and feature-extractor workloads, but becomes marginal for heavier detectors and especially for transformer-scale NLP unless aggressive tuning is applied.

Published guidance on precision and quantization is not fully uniform. One benchmark states that INCS only accepts FP16/FP32 ONNX to OpenVINO IR and reports no INT8 quantization path [2306.12093]. A later sustainability study, however, describes OpenVINO IR on INCS with graph-level optimizations including operator fusion, pruning, and quantization, and recommends INT8 quantization and operator fusion for improved performance [2507.23093]. Similar device-specific guidance converges on several operational practices despite that discrepancy: exploit FP16 when targeting Myriad X, warm up before timing, separate enqueue from dequeue when using NCAPI, overlap host-side work with device execution, match model structure to the VPU’s channel-block and scratch-pad behavior, and avoid architectures whose latency or memory behavior is incompatible with the device envelope [1810.04150] [2305.15422] [2409.12939] [2305.03739].

In aggregate, the literature places INCS at the intersection of embedded heterogeneous computing, low-power CNN inference, and hardware-aware model design. Its distinguishing properties are not absolute throughput leadership, but a combination of VPU-centric programmability, stable single-stream latency, support for a wide range of model families, and a software path that integrates classical CV, CNN inference, and hardware-aware NAS into a single edge deployment target [2306.12093] [2409.12939] [2305.03739].

Source: https://www.emergentmind.com/topics/intel-neural-compute-stick-incs