Performant Python pipelines without the Rust

Justin Joyce · 18 October 2026 · 6 min read

avro rust performance benchmarks

Correct at time of publishing. For up-to-date performance and feature information, see rugo.dev.

"Rewrite it in Rust" has become the default answer to a slow Python pipeline. We put that to the test, and Python won. Called from Python, rugo beat arrow-avro, Apache Arrow's own Rust Avro reader, in three of four races on one thread. On its defaults it was 2.3× to 4.5× faster than anything Rust managed.

You know how this goes. The pipeline gets slow, someone profiles it, and a Rust rewrite gets scoped. Six weeks later you own a second language, a second build system, a pile of FFI glue, and a codebase half your team can't review. The pipeline is faster. Was it the language, though, or the engine underneath it?

Usually it's the engine. To prove it, we raced rugo's Avro reader, called from Python, against the best Avro readers Rust has, called from Rust.

The contenders

arrow-avro is the one to beat. It decodes straight into columns, as rugo does, and in our runs it was 2.7× to 16× faster than the older apache-avro crate. We ran it twice: once on its defaults, and once tuned as far as we could push it, with 65,536-row batches and its deflate switched to zlib-rs, a faster Rust inflater than its default.

The test

One file: 200,000 rows of a 20-field event record, the mix you'd see in a Kafka topic or a log stream. Longs, doubles, strings, enums, nullable unions, timestamps, dates, a decimal, a fixed(16), a nested record, an array and a map. fastavro wrote it, so no reader is timed on its own output.

We wrote it twice, once with deflate (21.6 MB) and once uncompressed (38.2 MB), and read it two ways:

• all fields: every one of the 20.
• 3 fields: id, amount and country. rugo is given columns; arrow-avro a projection; apache-avro a reader schema of just those fields.

Every reader decoded the same in-memory bytes, so disk wasn't part of the timing. Each run was a fresh process on one thread. The readers took turns across five rounds, and the figures are medians. The machine is an 18-core Apple Silicon Mac.

One thread each

Median read time, shorter is faster:

Python won three of the four. The arrow-avro figures are its tuned build; on its defaults it was 15 ms slower on the deflate races and level on the uncompressed ones. apache-avro was never in it: it came last in every race, up to 23× slower than rugo.

Every core

One thread each keeps the race fair, but it isn't how anyone runs rugo. On its defaults, rugo spreads a file's blocks across every core. Neither Rust reader can do that for a single file, so this compares rugo, untouched, with arrow-avro at its best:

That's what you get without changing a setting.

Where we lost

One race went to Rust. Reading every field of the uncompressed file on one thread, arrow-avro took 40.6 ms and rugo took 47.9 ms, so arrow-avro was 18% faster.

We're showing it because it's the purest test of the decoder itself: no inflate, no skipping, just bytes into columns. It's also the one we most want to win.

Our best guess, not yet tested: rugo turns the nested record and the map into JSON text, which is extra work. arrow-avro builds native struct and map arrays instead. Reading the file without those two fields would tell us whether the decoder itself is behind.

It doesn't change the headline. Kafka Connect, Spark and Iceberg usually write Avro compressed, and on compressed files Python wins.

Python was never the slow part

Here's the bit the rewrite-it-in-Rust crowd skips: in a well-built library, Python isn't doing the work. rugo is a native C++ engine with a Python front door. You call it from Python, and the bytes never touch the interpreter. What makes it fast is the engine, and an engine can't tell which language called it:

• Compile once, run per block. rugo turns the Avro schema into a flat program of decode steps, then runs it over every block. No lookups per row.
• Straight into columns. Values land in typed columnar vectors, never as a Python object each. It's exactly what you'd write in Rust. You just don't have to write it.
• Inflate is the real fight. On compressed files, decompression is most of the work, so rugo uses libdeflate, a dedicated, heavily tuned inflate library. Choosing the right inflater matters more than choosing a language.
• Don't decode what you don't read. Fields you don't ask for are stepped over. On the uncompressed file, reading 3 of 20 fields takes about a third of the time of reading them all.
• Use the whole machine. A file's blocks are independent, so rugo spreads them across every core and still hands batches back in file order. Neither Rust reader does that for a single file.

None of this means Rust is slow; arrow-avro is excellent work. It means "fast" belongs to the engine, not to the language you call it from. If the engine is good, you can keep the language your team already writes.

What this leaves out

• One schema. A wide, flat record, or one that's mostly strings, could move the margins either way.
• One machine. Every figure here is from Apple Silicon. We haven't run x86 yet.
• Different outputs. rugo returns nested records and maps as JSON text; arrow-avro returns native struct and map arrays. Both end in columns, but they aren't the same columns.
• Other languages. We didn't test Go, Java or C++ readers. The ones we know of decode a row at a time, as apache-avro does.
• Reading the file. Every reader got the bytes already in memory. Reading from disk or object storage adds the same time to each.

Try it

pip install rugo is the whole install. There's no toolchain to set up, and no PyArrow or NumPy:

from rugo.avro import read_avro

with read_avro(
    "events.avro", columns=["id", "amount", "country"]
) as reader:
    for morsel in reader:
        print(morsel.num_rows, morsel.column("amount").to_pylist()[:3])

Reproducing this

Measured 18 October 2026 on an 18-core Apple Silicon Mac, against rugo 0.5.4 from PyPI on CPython 3.14, and arrow-avro 60.0.0 and apache-avro 0.22.0 built with Rust 1.96.1 (release profile, fat LTO). The tuned arrow-avro build switches flate2 to zlib-rs 0.6.5; the default build uses miniz_oxide. rugo runs with max_threads=1 for the one-thread races.

Each of the five rounds runs every reader in a fresh process, five reads each, with the order rotated each round so no reader always goes first. The figures are the median of the per-process medians.

The harness, the data generator and the Rust readers are in benchmarks: measure_avro_rust.py builds both Rust readers, writes the file and runs the race on your machine. We'd like to hear what you get, especially on x86.