Understanding Avro: the file format built for moving data

Justin Joyce · 21 October 2026 · 10 min read

avro file formats schema evolution python

Correct at time of publishing. For up-to-date performance and feature information, see rugo.dev.

The first rocket launch in our dataset, Sputnik-1, takes 234 bytes as a line of JSON and 96 bytes as Avro. Nothing in those 96 bytes says what a field is called or where it ends, and any program that holds the schema can still read it back exactly. That trade, a schema in exchange for bytes, is most of what Avro is.

Avro started in 2009 inside the Hadoop project, and it has kept the job it was built for: moving records between systems that are written, deployed and upgraded separately. It's what you meet in event streams, in message queues, and in the landing zone where raw data arrives before it's tidied into something else.

In an earlier post we put Avro next to Parquet and JSON Lines. This one opens Avro up. We'll take a single record, then a file, then a schema change, and look at the bytes each time. The data is the same 4,630 orbital launch attempts, and the file is in the rugo.dev repository if you want to follow along.

It starts with a schema

Every Avro file is described by a schema, and the schema is plain JSON. Here's the one for our launches, trimmed to its shape:

{
  "type": "record", "name": "Launch",
  "fields": [
    {"name": "Company",    "type": "string"},
    {"name": "Price",      "type": ["null", "double"]},
    {"name": "Lauched_at", "type": ["null", {"type": "long", "logicalType": "timestamp-micros"}]},
    {"name": "Rocket",     "type": {"type": "record", "name": "Rocket", "fields": [
      {"name": "Name",     "type": "string"},
      {"name": "Status",   "type": {"type": "enum", "name": "RocketStatus",
                                    "symbols": ["Active", "Retired"]}}
    ]}},
    {"name": "Mission",    "type": "string"}
  ]
}

There are two layers of types. The base layer is small: null, boolean, int, long, float, double, bytes and string, plus the containers record, enum, array, map, union and fixed. On top sit logical types: a name that says what a stored value means. A timestamp-micros is a long counting microseconds. A decimal is bytes. Readers that know the logical type hand you a timestamp or a decimal, and readers that don't still get the long or the bytes.

Two of those types carry more weight than they look. A union is how Avro says "or", and a union with null is how it says "optional": ["null", "double"] is a price that may be missing. A record can hold other records, so nested data keeps its shape instead of being flattened into Rocket_Name and Rocket_Status.

What 96 bytes say

Here is the first record in the file, cut out of the first block and read byte by byte.

Read down the table and the whole encoding is there. A string is its length, then its UTF-8 bytes. The 12 in front of "RVSN USSR" is 9, the length, written the way Avro writes every integer (the next section explains that). The union in Price is a single byte, 00, which names branch 0, the null, and has no value after it. In Lauched_at the branch is 02, branch 1, and a long follows. The enum is the position of its symbol: 02 is 1, which is "Retired".

What isn't there is as important. There are no field names, because the schema has them. There are no field numbers or type tags, which is where Avro parts ways with formats like Protocol Buffers: the fields appear in the order the schema lists them, and that order is the only index. And there are no delimiters. Every length comes first, so a reader never scans for a quote or a comma.

The cost of that is stated plainly in the spec: you can't read the data without the schema it was written with. Everything else about Avro, from the file header to schema registries, is a way of making sure the schema arrives with the bytes.

Small numbers stay small

Integers are where the saving is easiest to see. Avro writes every int and long with the same two ideas, and both appear in the record above.

Zigzag deals with the sign. A negative number written the usual way sets its top bits and needs every byte, so Avro interleaves the signs instead: 0, -1, 1, -2, 2 become 0, 1, 2, 3, 4. A number near zero stays near zero whichever side it's on. Varint then writes it seven bits at a time, with the top bit of each byte saying whether another byte follows. Anything from -64 to 63 fits in one byte, -8,192 to 8,191 in two, and so on, so a number only ever takes the bytes it needs.

That's why a string's length costs one byte, why the nullable price costs one byte when it's missing, and why a row with a lot of small counters and flags comes out so much smaller than its text. A double is the exception: it's always eight bytes, just as a float is always four.

A file is a header, then blocks

A single record isn't a file. The Avro container file wraps a stream of records so that it can be read, split and appended to.

The header opens with four bytes, Obj and a version, then a small map of metadata: the schema, under the key avro.schema, and the codec, under avro.codec. It ends with a 16-byte random sync marker. In our file the schema is 603 of the header's 658 bytes, and the header is 0.6% of the file, which is why carrying the schema with the data is affordable.

After the header come the blocks, and every block has the same four parts: how many records it holds, how many bytes it takes, the records themselves, and the sync marker again. fastavro, which wrote this file, fills a block with about 16 KB of records, compresses it, and starts the next. Our 4,630 launches land in 29 blocks of 109 to 172 records, each taking 3 to 5 KB on disk.

Compression happens one block at a time

The codec named in the header applies to every block. The spec requires two, none (which the spec spells null) and deflate, and allows others: Snappy, zstd, bzip2 and xz are the common ones.

Before any compression, the Avro file is already 2.4× smaller than the JSON Lines one: 463 KB against 1.09 MB. That's the encoding at work, keys and quotes and text numbers gone. Deflate then takes it another 4.1× down to the 113 KB file in the repository. Snappy gives up some of that size for speed, xz gives up speed for size, and deflate is the one every Avro library must be able to read, which is why it's the safe default for files other people will open.

Because each block is compressed on its own, the codec never has to look outside it. Any block can be inflated without its neighbours, and that's what makes the next two properties possible.

Splitting and appending

The sync marker is a boundary you can find without parsing. A reader dropped into the middle of a file scans forward until it sees the 16 bytes, and from there it's on a block edge and can read normally.

That makes an Avro file splittable. A query engine can cut a large file into byte ranges, give each range to a different worker, and trust every worker to find its first whole block. It also makes the file appendable. A writer reads the header for the schema, codec and sync marker, then adds new blocks after the last one. No footer has to be rewritten, because there isn't one.

Schema evolution

Here is the reason Avro is the format between services. Producers and consumers upgrade on different days, and the data written by one version has to be readable by another.

Avro's answer is to give a reader two schemas: the writer's, which came out of the file's header, and the reader's, which is the shape your code wants today. The reader resolves one against the other by field name.

The rules are short. A field the reader wants and the file has is read. A field the reader wants and the file doesn't have takes its declared default. A field the file has and the reader doesn't want is skipped. Numbers can widen, from int up to long, float and double, and a field can be renamed if the new schema lists the old name as an alias. What can't happen is a field with no default appearing from nowhere, which is why give every new field a default is the one rule to remember.

Nothing in the file changes. The same bytes serve every reader schema, and all the work happens when the file is read. In rugo it's one argument:

from rugo import avro

reader_schema = {
    "type": "record", "name": "Launch",
    "fields": [
        {"name": "Mission", "type": "string"},
        {"name": "Price",   "type": ["null", "double"], "default": None},
        {"name": "Crewed",  "type": "boolean", "default": False},  # not in the file
    ],
}

with avro.read_avro("space_missions.avro", reader_schema=reader_schema) as reader:
    for morsel in reader:
        print(morsel.column_names, morsel.column("Crewed").to_pylist()[:3])

The file was written before Crewed existed. All 4,630 rows come back with it set to False, and the fields the reader schema leaves out are dropped.

What Avro doesn't do

Everything above comes from storing rows. The cost is the flip side of it. An Avro block holds a record count and a byte size, and that's all it says about its contents. There are no minimums and maximums, no per-field offsets, and no index.

So a reader that wants three fields of eight still inflates every block and walks every record. A string can be hopped over by its length. A number can't, because its size is in its own top bits, so the reader reads until it finds the end. A good reader never turns the unwanted fields into values, and that's a large saving. But it can't skip a block, because nothing says the block can be skipped.

That is the line between Avro and Parquet, and it's a fair one. Avro is built for writing one record at a time and for reading whole records. When the question is "how many launches did SpaceX fly?" over a billion rows, a columnar format wins by not reading the answer's neighbours. When the job is carrying each launch from one system to the next, Avro wins by not caring.

Avro without a file

Most Avro that crosses a network isn't a file at all. A message on a queue is one record, and repeating a 603-byte schema in front of it would cost more than six times the record. So systems send just the record and put the schema somewhere both ends can find it.

The common pattern is a schema registry. A producer registers its schema and gets an ID back, then writes each message as a short prefix (in Confluent's wire format, a zero byte and a four-byte schema ID) followed by the Avro bytes. A consumer reads the ID, fetches the writer schema from the registry once, caches it, and resolves it against its own reader schema. It's the same two-schema idea as the file, with the registry standing in for the header. Avro also defines a self-contained single-object encoding, a marker and an eight-byte fingerprint of the schema ahead of the record, for the same purpose.

When to use Avro

• Records move between systems on different release cycles. Schema evolution is the feature.
• Records arrive one at a time and the file has to keep up. Appending a block is cheap, and there's no footer to protect.
• The data is nested or richly typed. Timestamps, decimals, enums, arrays and maps all have a home, and they arrive as what they are.
• You'll mostly read whole records. Row storage is an advantage there, not a handicap.

And when the data settles in storage and gets queried a few columns at a time, compact it into Parquet. The two formats hand off to each other: Avro at the edge, Parquet at rest.

Where rugo fits

In Python, the usual route into an Avro file is a reader that builds an object for every record. rugo reads Avro blocks into typed columns instead, and only the columns you ask for.

from rugo import avro

# Nested fields are addressed with a dot: "Rocket.Name"
with avro.read_avro("space_missions.avro", columns=["Company", "Rocket.Name", "Price"]) as reader:
    for morsel in reader:
        print(morsel.column_names)
        print(morsel.column("Rocket.Name").to_pylist()[:2])

That prints the three column names and ['Sputnik 8K71PS', 'Sputnik 8K71PS']. Avro has no statistics, so there's no predicates argument as there is for Parquet; you filter the values you get back. Logical types arrive decoded, enums as their symbols, and the same morsel feeds the Parquet writer when it's time to compact.

Reading every field of 200,000 rows, rugo is 7.8× faster than fastavro and 50× faster than Apache's own avro package. Called from Python it also beat Apache Arrow's Rust Avro reader in three of four races. The home page has the measurements, with versions and hardware.

Try it

pip install rugo is the whole install. The Avro notebook runs in your browser on the same 4,630 launches: columns, nested fields, reader schemas, and a conversion to Parquet.

Reproducing this

Every byte count comes from docs/data/space_missions.avro, parsed with nothing but the spec: the 658-byte header, the schema's 603 bytes, the 29 blocks and their record counts and sizes. The 96-byte record is the first record of the first block, inflated and split using the schema. The 234 bytes is the first line of space_missions.jsonl, which flattens the rocket into two fields where Avro nests it. The codec sizes come from writing the same 4,630 records with fastavro 1.13 at each codec's defaults, so the deflate figure (112,731 bytes) differs from the repository file (112,717) by a few bytes. The reader-schema example was run with rugo 0.5.4.

The diagrams are generated by tools/build_avro_diagrams.py, which parses the Avro file directly.