What is ONNX?

A model trained in PyTorch is, by default, a pile of Python code plus weights. To run it you need Python and PyTorch, which is fine on a training server and awkward in a phone app, a web browser, or a C# service. ONNX solves that by saving the model as a self-contained file in a standard format that many runtimes understand.

What is in an ONNX file

An .onnx file holds 2 things: the computation graph, step by step (multiply this by that, add, apply this function), and the trained weights. The steps come from a fixed, shared list of building blocks called operators. Because every tool agrees what each operator means, any runtime that implements them can run the model, with no trace of the code that trained it.

The project was started by Microsoft and Facebook in 2017 and is now an open standard with many companies behind it.

How you get one

Export from the framework you trained in:

import torch

model.eval()
example = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, (example,), "model.onnx", dynamo=True)

The exporter runs the model on the example input to record the graph. If the model should accept other input sizes, such as different batch sizes or text lengths, you say so with dynamic_shapes, or the file will only accept the size you exported with. Hugging Face's Optimum library wraps this for most transformer models.

Where it runs

ONNX Runtime, from Microsoft, is the main engine. It runs on Windows, Linux, macOS, Android and iOS, in the browser through WebAssembly and WebGPU, and from Python, C++, C#, Java and JavaScript. It can hand the work to whatever the device has: an NVIDIA GPU, an Intel or AMD chip, Apple's neural engine, or plain CPU.

That is the main reason to use ONNX: one exported file, many places to run it. It also often runs faster than the original framework, because the runtime fuses steps together and can quantizeQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. the weights to 8 bits.

When it fits

ONNX is a strong choice for small and medium models deployed widely: image classifiers, embedding models, speech models, rankers, anything that must run on a device or inside a non-Python service. For serving large language models on GPUs, specialised servers such as vLLMvLLMAn open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer. or TGIText Generation InferenceHugging Face’s optimized server for LLM deployment. usually do better, because they handle the KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. and batching of many conversations that a general runtime does not.