※ Kernwerk

What is ONNX? The format everything converts through, and almost nothing runs

September 15, 2026 · by Maxime Carriere

Almost every model that reaches an edge device passes through ONNX. It is where a model leaves the framework it was trained in and enters the toolchain that compiles it for one chip. Ours are no exception: the wildfire gate to a €27 board, the chest X-ray box to a $249 Jetson, both by the same road.

ONNX is the most important file format in edge AI, and almost nothing executes it. It is an interchange format, which means it states what has to be computed and in what order, and says nothing about how. A runtime executes. An interchange format only describes, so that a producer and a consumer written by different people, in different languages, for different hardware, can agree on a model without sharing any code.

ONNX is where your model changes hands, not where it lives.

A funnel. Four training frameworks across the top, PyTorch, TensorFlow, JAX and scikit-learn, with lines converging into a narrow box in the middle labelled ONNX, a graph, weights, an opset. From the bottom of that box, lines diverge to six vendor toolchains and the parts they target: TensorRT on NVIDIA Jetson, RKNN on Rockchip, TIDL on TI Sitara, eIQ on NXP i.MX, Hailo SDK on Hailo-8, OpenVINO on Intel.
Everything above the neck varies, everything below it varies, and the thin part in the middle is the only thing both sides agree on.

What it is, mechanically

An ONNX file holds three things.

Three labelled bands showing the contents of one ONNX file. VERSION: ai.onnx opset 17, IR version 8, glossed as which edition of the operator definitions. GRAPH: an input tensor frame of shape 1x1x96x96 feeding a chain of boxes, Conv, Relu, ten more pairs, GlobalAveragePool, Flatten, Gemm, producing an output score of shape 1x1, noted as 25 nodes and 5 distinct operators. WEIGHTS: four named tensors with their shapes and dtypes, totalling 24 initializers, 132,417 parameters and 517 KB at fp32.
One real file, the 96 pixel wildfire gate exported from PyTorch. Nothing in it says how any of it should be computed.

The file holds 132,417 parameters. The wildfire post reports 133,889 for the same model, and the difference is not an error in either. It is 1,472, which is exactly the sum of the eleven convolutions’ output channels: the exporter folded each batch normalisation into the convolution before it, replacing a per channel scale and shift with one fused bias. The graph contains no normalisation nodes at all. Trading two operators for one, at identical arithmetic, is what saying nothing about how buys.

  • The graph: what to compute, and in what order. Nodes are operators, Conv, Relu, Resize, MatMul. Edges are the tensors passing between them.
  • The weights: the trained values. One named tensor per parameter, each with a shape and a type, stored alongside the graph.
  • The version: which edition of the operator definitions applies. The specification calls it the operator set, or opset, and a file carries one number for it. Version 17 fixes what Resize means, which attributes it takes and how it behaves at the edges. An exporter picks one version to write, usually the newest it supports. An importer accepts a range, usually everything up to whatever it last updated for. Nothing makes the two meet in the middle.

Nothing in the file says how any of it should be computed. An importer may fuse several operators into one kernel, run independent branches in whatever order the data dependencies allow, and choose which unit on the chip executes each one. What it may not do is change the dependencies: the graph fixes what feeds what, and a node cannot run before the node producing its input. The file is a requirement list: these operators, at this edition of their definitions. Whether it can be satisfied is a property of the toolchain reading it, not of the model.

Why edge AI needs it more than the cloud does

In the cloud the question barely comes up: the training framework and the serving stack are usually the same software on the same kind of machine. At the edge they never are. A model trained on a workstation GPU has to run on a part costing a few euros, through a toolchain written by the company selling the part.

Call N the frameworks and M the chip toolchains. Wire every pair directly and somebody maintains N times M converters, with every new chip adding another N. Put one format in the middle and the count is N plus M, one exporter per framework and one importer per vendor.

Two wiring diagrams. On the left, one converter per pair: four framework dots each joined to all five toolchain dots, twenty lines, labelled 4 times 5 equals 20 here, 10 times 40 equals 400 in the market. On the right, one format in the middle: the same nine dots each joined to a single box marked ONNX, nine lines, labelled 4 plus 5 equals 9 here, 10 plus 40 equals 50 in the market.
The same nine participants, wired both ways. The format does not make any single conversion better, it makes the number of conversions smaller.

Which is what happened. Walk the silicon band of the consolidation map and every toolchain on it speaks ONNX at the entrance:

  • NVIDIA TensorRT parses ONNX and builds an engine from it, which is the subject of an earlier post.
  • Rockchip RKNN converts ONNX to an RKNN blob.
  • TI TIDL imports ONNX as one of its two front ends.
  • NXP eIQ takes ONNX into its toolchain for i.MX parts.
  • Hailo’s compiler takes ONNX and produces an HEF binary.
  • Qualcomm, Renesas, STMicroelectronics all accept it, through their own tools, at their own version levels.

The shape has a name in networking. Many link technologies below, many applications above, and exactly one network layer between them: IP. Neither side coordinates with the other, because each only has to conform to the middle. The middle pays for that by staying small and changing slowly, at whatever speed both sides will follow.

ONNX holds the same position between models and silicon, under the same bargain: a few hundred operators, added slowly, removed almost never.

Where the contract breaks

Three failure modes, in rising order of how much time they cost.

Version mismatch. The exporter writes version 18, the importer supports up to 13. The direction matters: older files are usually safe, because importers keep the old operator definitions, and newer ones are not. The tool either refuses the file, or reads an operator by the older definition it knows and returns numbers that are close but not the same.

A different graph from the same model. Version mismatch is a disagreement about what the operators mean. This one leaves the vocabulary intact and changes what the file says, because the exporter is itself a program that changes between releases: it captures the model differently, maps PyTorch operations onto different ONNX nodes, and folds away fewer constants when its shape inference resolves less. Whatever it cannot resolve stays in the graph as Shape, Gather, Concat, Reshape, recomputing a constant on every frame. Pin the exporter version and diff the node count in CI.

The operator that exists on paper. The worst one. The operator is in the specification, the exporter emits it, the importer parses it, and the accelerator has no implementation, so the toolchain falls back to the CPU for that node on every frame. No error, no warning, and a 30 ms inference becomes 300 ms, which is why the compilation post measured rather than trusted the toolchain. The usual suspects are Resize with an awkward coordinate transform mode, NonMaxSuppression in a detector head, GridSample in anything that warps, and any custom operator out of a research repository. The gate model in the figure above needs five operators and every importer has all five, which is why it ports cleanly. A detector asks for more.

The contenders

ONNX is not unchallenged, and the challengers are structural rather than technical. Each has a domain where it is the normal choice, as of September 2026.

  • ONNX is the normal choice for classic vision on an accelerator: convolutional networks, detectors and classifiers, going into a vendor toolchain. Most of what ships on edge parts today.
  • ExecuTorch, from Meta, is the normal choice when the team is PyTorch end to end and the target is a phone or a Linux class device. It exports a PyTorch program and runs it with a PyTorch runtime, with no interchange step in the middle, which makes it the most serious challenger: it removes the border rather than competing to be one.
  • LiteRT, formerly TensorFlow Lite, from Google, is the normal choice on microcontrollers and on Android. The ESP32 gate runs through TFLite Micro, and at that tier ONNX is frequently not in the picture at all.
  • GGUF is the normal choice for language models running on the device, and it comes as a pair with its runtime. llama.cpp, the project that began as a plain C++ implementation of Meta’s Llama models and now runs most of the open ones, defines the format, writes it and loads it. Unlike ONNX, the format and the engine are one project, and the specification is the implementation. The on-device language model wave went here rather than through ONNX, and nothing suggests that traffic is coming back.
  • Vendor-native endpoints, a TensorRT engine, a Hailo HEF, an RKNN blob, a Core ML package, are what every path above eventually produces. Not competitors to the waist but where it ends, each built for one vendor’s silicon and portable nowhere.

One difference matters more than the technical ones. ONNX is governed by the Linux Foundation. ExecuTorch is Meta’s, LiteRT is Google’s, GGUF belongs to a project, and every vendor endpoint belongs to the company selling the chip. The neutral layer in this stack is the one with no giant behind it, which is the consolidation argument one layer up: the independent piece survives exactly as long as no one has an interest in owning it.

Where we fit

Every failure above is a decision taken by default. Almost nobody chooses an export version, freezes the shapes their target wants frozen, or checks which operators their accelerator can actually run. The defaults are taken, and the answer arrives at integration time.

Our work sits on both sides of the export.

Three boxes in a row joined by arrows: your trained model, ONNX, and the vendor toolchain and the part. Under the first arrow, before the export: pick the version the importer supports, freeze the shapes it wants frozen, swap an operator it cannot accelerate. Under the second arrow, after the export: read the operator histogram, find what fell back to the CPU, measure what the conversion cost.
The boxes are somebody else's software. The arrows are where the decisions get made, and where they are usually left to a default.
  • Before it, choosing the version and the shape policy the target toolchain handles best, and where the model asks for an operator that part cannot accelerate, changing the model rather than the flags.
  • After it, reading what came out: the operator histogram, what fell back to the CPU, how much shape arithmetic survived folding, and what the conversion cost in accuracy once the weights are quantised.

The export is also where compression starts, which is why we do not treat the two as separate jobs.

Shipping a model across this border, onto a part you have already chosen? We do the crossing: compressed onto the part you can afford, and sealed so it cannot be lifted back off the board. Talk to us.