Avro vs Parquet vs JSON Lines: one dataset, three very different files

Justin Joyce · 20 October 2026 · 9 min read

avro parquet jsonl file formats

Correct at time of publishing. For up-to-date performance and feature information, see rugo.dev.

The same 4,630 rows take 1.09 MB as JSON Lines, 113 KB as Avro and 93 KB as Parquet. Size is the least interesting difference. What matters is how much of each file a reader can skip, and that decides where each format belongs.

Ask which format to use and you'll get a one-line answer: Parquet for analytics, Avro for streaming, JSON Lines for logs. All three are true, and none of them helps until you know why. The reasons are in the bytes, so we'll open the same dataset in all three formats and look.

The data is 4,630 orbital launch attempts, from Sputnik onwards: company, rocket, price, launch time, outcome. It's small enough to inspect by eye, and every file here is in the rugo.dev repository if you want to follow along.

The one idea: rows or columns

A table is two-dimensional. A file is one long run of bytes. Every format has to pick an order for the cells, and there are only two sensible choices: write one whole record after another, or write one whole column after another.

JSON Lines and Avro are row-oriented. Parquet is column-oriented. Nearly everything else in this post follows from that choice, and from what each format adds to make its choice workable.

JSON Lines: one record per line

JSON Lines (also called NDJSON) is as simple as a format gets: one JSON object per line, separated by newlines.

Its strengths come from that simplicity. Any language can write it. You can open it in a text editor, search it with grep, and add to it with nothing more than a file write. A half-written file is still a valid file up to its last complete line. There's nothing to install and nothing to configure, which is why it's the default for logs and API exports.

The costs come from the same place. Every line repeats every key, so the field names are written 4,630 times. Numbers and timestamps are text that has to be parsed on every read. There is no schema, so the type of Price is whatever the first rows suggest, and a reader has to find the start of each record by scanning for newlines. To get at one field you scan the whole file.

Avro: a header, then blocks

Avro keeps the row orientation and makes everything about it explicit. The file opens with a header holding the schema as JSON, then holds the records in blocks.

The schema is the point of Avro. It travels with the data, so the file explains itself. It supports rich types: timestamps, decimals, enums, nested records, arrays and maps. Records are packed back to back with no keys and each block is compressed, so this file comes out nine times smaller than the JSON Lines one. And because readers and writers each declare a schema, they can differ: add a field with a default value and old files still read, which is why Avro is so common between services that are deployed on different days.

Blocks are the other half. Each one is compressed on its own and ends with the sync marker, so a reader can drop in at any block boundary. That makes an Avro file splittable across workers, and cheap to append to: you add another block.

What Avro doesn't keep is anything about the contents. A block records how many records it holds and how many bytes long it is, and nothing else. There are no per-column statistics to tell a reader a block can't match, so a read inflates every block and walks every record, skipping over the fields it doesn't want.

Parquet: column chunks and a footer

Parquet turns the layout on its side. Rows are grouped into row groups, and inside each row group every column is stored as its own contiguous chunk. The file ends with a footer that says where everything is.

Two things fall out of that. The first is compression. Put a thousand company names next to each other and they compress far better than the same names scattered between prices and timestamps, and a column of repeated values shrinks further still through dictionary encoding. Each column gets the encoding that suits it. That's how this file ends up the smallest of the three.

The second is skipping. The footer records the minimum and maximum of every column chunk, plus a null count. A reader that wants launches priced under $60M can skip any row group whose cheapest launch already costs more, without opening it. A reader that wants three columns of eight opens three chunks. Skipping is built into the layout.

The price is that Parquet files are written, not appended to. The footer is written last and the file can't be read without it, so a half-written Parquet file is unreadable and adding a record means writing a new file. Writers buffer a whole row group in memory before they emit it. Parquet suits data you write in batches and read many times.

What a read actually touches

Here's the same question asked of all three files: the company, price and outcome of every SpaceX launch. It needs three of the eight columns.

JSON Lines and Avro have to go through the whole file. Parquet reads 7,254 bytes of 93,169, because the footer points straight at the three chunks that matter. On a file this size nobody would notice. On a 2 GB file with 110 columns, reading a handful of them is a very different job in each format.

Compression doesn't change the picture for JSON Lines. Gzip takes this file to 81 KB, but a reader still has to inflate all of it to reach any one field.

Side by side

Which one should you use?

Ask how the data will be written, then how it will be read.

• Records arrive one at a time, and people need to look at them: JSON Lines. Logs are the classic case. The format costs you in size and read speed, and gives you grep.
• Records move between systems, and the shape will change: Avro. A schema that travels with the data and evolves safely is worth more than a smaller file.
• Data is written in batches and read many times, a few columns at a time: Parquet. Each read pays only for what it uses.

The mistake is treating this as one decision for the whole pipeline. Most pipelines use two of the three, because writing and reading want opposite things.

A row format at the edge takes records as they happen. A batch job compacts them into Parquet. Everything downstream reads Parquet. The formats don't compete; they hand off to each other.

Where rugo fits

That hand-off is where Python code spends its life, and it's why rugo reads all three formats through one API. You pass a path or bytes, name the columns you want and, where the format allows it, the predicates to filter by. What comes back is the same columnar batch whichever format it came from.

from rugo import avro, jsonl, parquet

want = ["Company", "Price", "Mission_Status"]
spacex = [("Company", "==", "SpaceX")]

# Parquet: unwanted chunks and row groups are never read
with parquet.read_parquet("space_missions.parquet", columns=want, predicates=spacex) as reader:
    print(sum(m.num_rows for m in reader))

# JSON Lines: unwanted fields are never turned into values
with jsonl.read_jsonl(
    "space_missions.jsonl", columns=want, predicates=spacex,
    explicit_schema={"Price": "FLOAT64"},
) as reader:
    print(sum(m.num_rows for m in reader))

# Avro: same columns; Avro has no statistics, so filter the values
with avro.read_avro("space_missions.avro", columns=want) as reader:
    print(sum(c == "SpaceX" for m in reader for c in m.column("Company").to_pylist()))

All three print 182. Moving data along the pipeline in the previous section is a read and a write, because every reader and writer shares the same batch:

with jsonl.read_jsonl("events.jsonl") as reader:
    with open("events.parquet", "wb") as f:
        with parquet.open_parquet_writer(f.write) as writer:
            for morsel in reader:
                writer.write_row_group(morsel)

rugo is built on the idea at the centre of this post: the fastest bytes are the ones you never read. That's why its numbers look the way they do:

• JSON Lines: on a 2.0 GB file of a million rows and 110 columns, rugo is 16× faster than PyArrow reading one column, and 23× faster with a filter that lets about 10% of rows through.
• Avro: 7.8× faster than fastavro and 50× faster than Apache's own avro package reading every field of 200,000 rows. Called from Python, it also beat Apache Arrow's Rust Avro reader in three of four races.
• Parquet: cold-starts 1.5–2.2× faster than PyArrow, DuckDB or Polars, from an install of about 8 MB with no PyArrow, pandas or NumPy underneath it.

The home page has the measurements behind each figure, with versions and hardware.

Try it

pip install rugo is the whole install. Each format has a notebook that runs in your browser on the same 4,630 launches used here: Parquet, Avro and JSON Lines.

Reproducing this

File sizes are the bytes on disk of space_missions.jsonl, .avro (deflate) and .parquet (zstd) in docs/data. The Avro file stores the rocket as a nested record; the JSON Lines and Parquet files flatten it, and all three hold the same 4,630 launches. The Parquet byte counts in the reading diagram are each column chunk's compressed size and the footer size from the file's own metadata, and the Avro block count is from its block headers. The 182 comes from running the code above with rugo 0.5.4.

The diagrams are generated by tools/build_format_diagrams.py, which reads the file sizes from the data files directly.