# What is ONNX?

An open file format for trained models, so a model built in one framework (such as PyTorch) can run in another runtime, on another device or language, without the original code.

- Page: https://tecyfy.com/concepts/onnx
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Open Neural Network Exchange, ONNX Runtime

A model trained in PyTorch is, by default, a pile of Python code plus weights. To run it you need Python and PyTorch, which is fine on a training server and awkward in a phone app, a web browser, or a C# service. ONNX solves that by saving the model as a self-contained file in a standard format that many runtimes understand.

## What is in an ONNX file

An `.onnx` file holds 2 things: the computation graph, step by step (multiply this by that, add, apply this function), and the trained weights. The steps come from a fixed, shared list of building blocks called operators. Because every tool agrees what each operator means, any runtime that implements them can run the model, with no trace of the code that trained it.

The project was started by Microsoft and Facebook in 2017 and is now an open standard with many companies behind it.

## How you get one

Export from the framework you trained in:

```python
import torch

model.eval()
example = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, (example,), "model.onnx", dynamo=True)
```

The exporter runs the model on the example input to record the graph. If the model should accept other input sizes, such as different batch sizes or text lengths, you say so with `dynamic_shapes`, or the file will only accept the size you exported with. Hugging Face's Optimum library wraps this for most transformer models.

## Where it runs

ONNX Runtime, from Microsoft, is the main engine. It runs on Windows, Linux, macOS, Android and iOS, in the browser through WebAssembly and WebGPU, and from Python, C++, C#, Java and JavaScript. It can hand the work to whatever the device has: an NVIDIA GPU, an Intel or AMD chip, Apple's neural engine, or plain CPU.

That is the main reason to use ONNX: one exported file, many places to run it. It also often runs faster than the original framework, because the runtime fuses steps together and can [quantize](https://tecyfy.com/concepts/quantization) the weights to 8 bits.

## When it fits

ONNX is a strong choice for small and medium models deployed widely: image classifiers, embedding models, speech models, rankers, anything that must run on a device or inside a non-Python service. For serving large language models on GPUs, specialised servers such as [vLLM](https://tecyfy.com/concepts/vllm) or [TGI](https://tecyfy.com/concepts#text-generation-inference) usually do better, because they handle the [KV cache](https://tecyfy.com/concepts/kv-cache) and batching of many conversations that a general runtime does not.

## Related

- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.
- [Inference Latency](https://tecyfy.com/concepts/inference-latency): The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).
- [Neural Network](https://tecyfy.com/concepts#neural-network): A computing system inspired by biological neural networks that learn patterns from data

## Sources

- [ONNX](https://onnx.ai/)
- [ONNX Runtime](https://onnxruntime.ai/docs/)
- [torch.onnx](https://docs.pytorch.org/docs/stable/onnx.html)