What is an NPU? The chip that can only do one thing

An NPU, or neural processing unit, is a processor specialised for neural networks. Unlike a CPU, it accelerates a limited set of operations rather than running arbitrary programs. It usually sits on the same chip as the CPU and takes over the model’s arithmetic, so the model runs faster, and usually on less power, than it would on the CPU.

Its speed is the first number on almost every edge board’s datasheet, in TOPS, trillions of operations per second: 6 TOPS, 13 TOPS, 26 TOPS. That number tells you how fast the NPU can multiply. It does not tell you how fast your model will run.

Inside an NPU

A CPU can run any program. An NPU runs a short list of operations, mostly multiply and add, and that is exactly why it is fast.

One operation, repeated

Every layer of a neural network comes down to the same step: multiply an input by a weight and add the product to a running sum, y = y + w × x. That step is a multiply-accumulate, or MAC. A convolution or a transformer layer is millions of MACs arranged as a matrix multiplication, in the same pattern on every frame. MobileNet v1, an image classifier used below, needs 569 million of them per image.

Four parts, and little else

An NPU is hardware built around that one step:

A grid of MAC units. Thousands of multipliers working in lockstep, each doing one multiply-accumulate per clock.
On-chip memory, SRAM, beside the grid. It keeps weights and activations close, and the compiler decides what goes in it, instead of a cache guessing.
Small fixed units after the grid. They apply activations, pooling and the rescaling of 8-bit results between layers, so values do not leave the chip between steps.
A controller with a fixed schedule. The compiler plans every step before the model runs. No branch prediction, no speculative execution, and few data types, usually 8-bit integers.

Three chip outlines side by side, drawn as schematics rather than to scale, with silicon coloured by what it is spent on: control, compute and memory. The CPU is mostly control logic and cache around a few large cores. The GPU is an array of small cores in blocks, each block sharing one strip of control. The NPU is almost entirely a grid of multiply-accumulate units beside a block of on-chip memory, with a small controller; arrows along one row and one column show each value reused across the grid rather than read again from memory. Below the NPU, dashed outlines name what it leaves out: branch prediction, out-of-order execution, 64-bit floating point and hardware-managed caches. — Each chip is coloured by what its silicon is spent on, and the dashed outlines are what the NPU leaves out.

In one line each:

CPU: runs any program. You write the code.
GPU: runs the same instruction on thousands of values at once. You write the code, in CUDA or similar.
NPU: runs a neural network’s arithmetic, mostly matrix multiplies, at low precision. You write no code: the vendor’s compiler turns your model file into its schedule.

What moves to the NPU

An NPU does not take over the program. It takes over the model’s arithmetic, and the clearest way to see what that means is one model on one chip. NXP times exactly that in the machine learning guide for its i.MX 8M Plus, updated in September 2026: MobileNet v1, an 8-bit image classifier, on the CPU and on the NPU.

Panel A, what is handed over, drawn as three boxes on an NXP i.MX 8M Plus: the CPU, the shared DRAM, and the NPU. The CPU holds the model file, 28 layers and 4.2 million weights, and the runtime with its NPU compiler. Step 1, once at load: the compiler writes an NPU program, the 28 layers compiled, and the weights, 4.2 million 8-bit values or about 4 MB, into shared DRAM. Step 2, every frame: the CPU writes the resized input image, 224 by 224 by 3 bytes or 150 KB. Step 3: the NPU reads the program, weights and input, streams them through its on-chip SRAM tile by tile, and runs 569 million multiply-accumulates. Step 4: the NPU writes 1,001 scores, about 1 KB, and the CPU picks the top 5. Camera, resize, top-5 labels, the application and starting each inference stay on the CPU. A note says a PCIe card such as Hailo-8 differs: the weights are copied into the card once and every input and output crosses PCIe. Panel B, speed of the model alone per image, from NXP's guide: 39.3 ms on the CPU, past a dashed line marking one frame at 30 frames per second, so about 25 frames per second; 2.6 ms on the NPU, fifteen times faster. — Sizes from the model itself; timings are NXP's reported averages, CPU thread count not stated, and MobileNet v1 is close to an NPU's best case.

In short, the steps in panel A:

Once, at load: the CPU compiles the 28 layers into an NPU program and stores the 4 MB of weights where the NPU can read them. The first inference is slower when the runtime compiles on demand; some runtimes cache the compiled model.
Every frame: the CPU hands over a 150 KB image; the NPU runs 569 million multiply-accumulates and hands back 1,001 scores.

The camera, the resize, the top-5 and the application stay on the CPU. Here the two share DRAM; a PCIe card such as Hailo-8 copies the weights in once and sends every frame over PCIe.

Panel B is the payoff: NXP reports 39.3 ms on the CPU and 2.6 ms on the NPU, figures we have not reproduced. A frame at 30 fps lasts 33 ms, so the CPU alone drops frames, while the NPU leaves most of the frame free. It also shows what TOPS is worth: 2.6 ms works out to about 0.44 TOPS, a fifth of the rated 2.3, on a best-case model.

It does not always go this well. In a 2022 study on the same chip, a licence-plate pipeline ran slower with its detector on the NPU, 1.18 frames per second against 1.99, though it freed the CPU. The authors do not say why; the next two sections give the usual reasons.

Why it is fast

The speed comes from not paying for anything but the arithmetic. Mark Horowitz’s 2014 table of energy costs, from the International Solid-State Circuits Conference (ISSCC), explains the whole design.

A horizontal bar chart on a logarithmic scale of energy per operation in picojoules, 45 nm at 0.9 volts. An 8-bit integer multiply costs 0.2 pJ. A 32-bit floating-point multiply costs 3.7 pJ. Reading 64 bits from an 8 KB on-chip memory costs 10 pJ. One scalar add instruction on a simple CPU, with fetch, decode and control, costs about 70 pJ. Reading 64 bits from a 1 MB cache costs 100 pJ. Reading 64 bits from DRAM costs 1,300 to 2,600 pJ. Bars are coloured as in the first figure: compute, control and memory. Three are tagged with the NPU design choice that answers them: low precision for the floating-point multiply, a fixed schedule for the instruction overhead, reuse on chip for the DRAM read. — Horowitz, ISSCC 2014, 45 nm at 0.9 V. Newer processes lower every bar, the arithmetic faster than the memory, so the gap has not closed since. The multiply is the cheapest thing on the chart.

Each expensive bar has a design choice answering it.

No instruction per operation. A simple CPU spends about 70 pJ on one scalar add instruction; the addition itself is 0.1 pJ. An NPU’s schedule is compiled once, and the grid runs for thousands of cycles without asking what to do next.
Low precision. An 8-bit integer multiply costs 0.2 pJ against 3.7 pJ for a 32-bit float, and each value is a quarter of the bytes to move. That is why quantization comes before the NPU, not after it.
Reuse, at every level. One byte from DRAM costs as much as 800 to 1,600 8-bit multiplies, and even a read from small on-chip memory costs more than the multiply it feeds. So the grid passes or broadcasts each value across many multipliers: a number read once feeds a whole row, 64 on the AM68A. Convolution, Horowitz noted, offers about the thousand operations per DRAM fetch this needs.

Why it breaks

The narrowness that makes an NPU fast is also where it fails. It executes a list of operators, and nothing else. When the compiler meets one that is not on the list, the model is split.

The supported parts run on the NPU. The unsupported operator runs on the CPU, often in floating point. At each boundary the runtime stops the NPU, hands the activations over, converts them to the CPU’s layout and precision, waits for the CPU, and converts them back. The operator itself may be cheap. The handover is not, and it happens on every frame.

Two lanes, NPU above and CPU below. A model graph runs left to right on the NPU lane: a block of convolutions, a dashed gap where one operator is missing, then more convolutions. At the gap the data drops to the CPU lane, labelled sync, relayout and int8 to fp32; the CPU runs the missing operator, GridSample; then the data climbs back, labelled fp32 to int8, relayout and resume. Below, one frame's timeline shows the NPU working, then idle during the handover out, the CPU compute and the handover back, then working again. — Illustrative, not measured. The operator the NPU lacks, here GridSample, is often the smallest node in the graph. Pipelining frames can hide the detour from throughput, never from the latency of each frame.

The usual suspects are the ones the ONNX post listed: Resize, GridSample, non-maximum suppression, custom operators. And the vendors do not even fail the same way: some quietly move the operator to the CPU, some refuse to convert the model. The table further down shows each one, from its own documentation.

Choosing a board by its NPU

The datasheet number answers the question nobody should be asking. Five parts that often share a shortlist, as their vendors describe them in October 2026, not measured by us. TOPS are peaks, every MAC busy on every cycle, which small layers never manage and a language model generating text, waiting on memory, never comes close to. Several standalone NPU makers have also been bought, Hailo, Kinara and Untether AI’s team among them, as the consolidation post traced, so their toolchains’ roadmaps now belong to chipmakers.

Part	Accelerator, vendor peak	Unsupported operator
Rockchip RK3588	NPU, 6 TOPS: three 2-TOPS cores, one used by default	CPU if RKNN has a kernel, else conversion fails
NXP i.MX 8M Plus	NPU, 2.3 TOPS; compiles on first run, seconds unless cached	stays on the Cortex-A53 cores
TI AM68A	matrix accelerator, 8 TOPS	TIDL partitions it onto the Arm cores
Hailo-8, an M.2 or PCIe card beside a host CPU	NPU, 26 TOPS	parser rejects it: cut the graph or change the model
NVIDIA Jetson Orin Nano Super	no NPU; GPU, 33 TOPS dense (67 sparse) at top power mode	a TensorRT plugin you write, or ONNX Runtime on CUDA or CPU

None of these numbers says how fast your model will run; only compiling and timing it on the board does. A part with a third of the TOPS and full coverage of your graph usually beats one with three times the TOPS and an operator on the CPU.

Nor do they say what your model will answer. Each toolchain rounds and accumulates in its own order, and when we ran one language model on two devices, the same weights changed the final answer on 7% of maths problems, and on 53% once the weights went to 8 bits.

Where we fit

Answering those questions on your model, rather than on a benchmark, is the work we do.

Three boxes joined by arrows. You send: your trained model, a test set and metric, your board list. Kernwerk does: compile per toolchain, measure on the board, fix what can be fixed. You get, per board, an illustrative row of four boards labelled A to D with coloured dots: operators all on the NPU or one on the CPU or the parser stops; latency fits or tight; accuracy holds or loses 3 points; and a verdict of support, fix one op, recalibrate, or rework or decline. A dashed loop from the report back to Kernwerk reads: a vendor ships a new toolchain, the matrix runs again. Footer: the decision it buys, which board to build on, which boards to support, which to decline. — Illustrative: the shape of the report, not a result.

You get, per board, the operators on and off the accelerator, end-to-end latency per frame, CPU load, accuracy on your own test set, and the model change that closes the gap, if one exists.
Measured on the board itself: on our kit where we have one, and on yours, shipped or over remote access, where we do not.

What it buys is a decision: which board to build on, or, if you sell the model, which boards to support, which to decline, and whether one model change beats a variant per vendor. When a vendor ships a new toolchain, we rerun the matrix.

We do not sell silicon and no chipmaker owns us, so the answer is allowed to be somebody else’s chip, or the cheaper board.

If the NPU on your board sits idle while the CPU does the work, that gap is the job. Send us your model and the boards on your list, and we will tell you which ones it runs well on, and why the others do not. Talk to us.