iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

ONNX Export

By · Reviewed & last updated · Part of PyTorch Tutorial

ONNX is an open format that lets you train in PyTorch and serve almost anywhere — ONNX Runtime, TensorRT, OpenVINO, CoreML, browser via ORT-Web. It is the right export target when you want broad runtime support and the smallest possible deployment surface, or you need a non-PyTorch C++/Rust integration.

Export, validate, run, and quantise an ONNX model

EXAMPLE
import torch
import torch.nn as nn

class TinyMLP(nn.Module):
    def __init__(self, in_dim=10, hidden=32, out_dim=2):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden)
        self.fc2 = nn.Linear(hidden, out_dim)
    def forward(self, x):
        return self.fc2(torch.relu(self.fc1(x)))

model = TinyMLP().eval()
example = torch.randn(1, 10)

# 1) Export — use a dynamic batch axis so the runtime can serve batches > 1
torch.onnx.export(
    model, example, 'tinymlp.onnx',
    input_names=['x'],
    output_names=['logits'],
    dynamic_axes={'x': {0: 'batch'}, 'logits': {0: 'batch'}},
    opset_version=17,
    do_constant_folding=True,
)

# 2) Validate the file with the onnx package
import onnx
onnx_model = onnx.load('tinymlp.onnx')
onnx.checker.check_model(onnx_model)
print(onnx.helper.printable_graph(onnx_model.graph)[:500])

# 3) Run it via ONNX Runtime (CPU)
# pip install onnxruntime
import onnxruntime as ort
import numpy as np
sess = ort.InferenceSession('tinymlp.onnx', providers=['CPUExecutionProvider'])
out = sess.run(['logits'], {'x': np.random.randn(8, 10).astype('float32')})
print('output shape:', out[0].shape)

# 4) Compare to PyTorch output (tolerance check)
with torch.no_grad():
    torch_out = model(torch.from_numpy(np.random.randn(8, 10).astype('float32'))).numpy()
print('mean abs diff:', np.mean(np.abs(out[0] - torch_out)))   # should be very small

# 5) Dynamic quantisation — int8 weights, fp32 activations. CPU 2-4x speedup.
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic('tinymlp.onnx', 'tinymlp_int8.onnx', weight_type=QuantType.QUInt8)

# 6) Verify the quantised model still answers correctly
sess_q = ort.InferenceSession('tinymlp_int8.onnx', providers=['CPUExecutionProvider'])
out_q = sess_q.run(['logits'], {'x': np.zeros((1, 10), dtype='float32')})
print('int8 output:', out_q[0])

# 7) Where ONNX shines:
#   - mobile (CoreML/iOS, NNAPI/Android via ORT)
#   - browser (onnxruntime-web running WASM/WebGPU)
#   - C++/Rust/Java servers without a Python dependency
#   - inference acceleration via TensorRT or OpenVINO

Why it matters

Always export with a dynamic batch axis. Static-batch ONNX models force the runtime to either pad single requests or refuse batch inference, both of which kill throughput. The two extra lines of dynamic_axes config are the cheapest performance win in the pipeline.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
import torch
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, dummy, 'model.onnx', input_names=['x'], output_names=['logits'])
Try it Yourself »

Discussion

Loading…