Read and write Parquet, CSV, and JSONL. Read Avro.
No PyArrow. No Pandas. No NumPy.

Traditional libraries optimise decoding speed. Rugo optimises reading less data in the first place — pushing column projection and predicate pushdown ahead of decode, so data you don't want is never touched. Extracted from years of work on the Opteryx SQL engine.

Performance doesn't only come from decoding faster.

Often it comes from avoiding decoding data you don't need in the first place. Rugo was built as the file layer for the Opteryx SQL engine, where the goals were simple: read as little data as possible, hold as little in memory as possible. Column projection and row-group pruning happen before any decoding — so the columns and rows a query doesn't need are never paid for. The result is a library that gets faster the more selective your query is, stays tiny on disk, and carries no PyArrow, Pandas, or NumPy into your environment.

The gap is projection, not decode speed.

Rugo wins by decoding less — up to 23× faster than PyArrow on selective JSONL reads. The more columns and rows a query lets rugo skip, the bigger the win, and even when it has to read everything it is still 2.6× faster.

JSONL — 1M rows, 110 cols, 2.0 GB

This is the case that shows the architecture. PyArrow reads and decodes every column on every query regardless of projection or filter, so asking for one column costs it the same as asking for all 110.

CSV — wide file, 50 cols, 200k rows, 83 MB

On CSV, rugo goes toe-to-toe with PyArrow — within a few of milliseconds on every query shape, and dead level once a predicate drops rows before the typed column build. All from a package a fraction of the size.

Avro — 200k rows, 20 fields, deflate, 21.6 MB

Against the two Python Avro readers: fastavro, and avro, the Apache project's reference implementation.

A different niche, not a smaller PyArrow.

Rugo is the file layer for applications that read data selectively rather than scan it wholesale — places where footprint, cold-start time, and memory matter as much as throughput.

One install, four formats.

Rugo reads and writes Parquet, CSV, and JSONL, and reads Avro. The API is the same shape across all four: pass a path or bytes, get columnar data back.

from rugo import parquet

# Read — context manager, yields Morsels
with parquet.read_parquet("planets.parquet", columns=["id", "name"], predicates=[("id", ">", 4)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# Metadata — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows, [c.name for c in meta.schema_columns])

# Write
data = parquet.write_parquet(morsel, compression="zstd")

from rugo import csv

# Same shape — context manager, yields Morsels
with csv.read_csv("data.csv", columns=["name"], predicates=[("age", ">", 30)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

meta = csv.read_metadata("data.csv")
data = csv.write_csv(morsel)

from rugo import jsonl

# Same shape again
with jsonl.read_jsonl("events.jsonl", columns=["id", "status"], predicates=[("status", "==", "active")]) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

meta = jsonl.read_metadata("events.jsonl")
data = jsonl.write_jsonl(morsel)

from rugo import avro

# Same shape — reader only; rugo doesn't write Avro
with avro.read_avro("events.avro", columns=["id", "device.os"]) as reader:
    for morsel in reader:
        print(morsel.column("device.os").to_pylist())

meta = avro.read_metadata("events.avro")   # writer schema + codec

# Look at a file
rugo info events.parquet              # rows, columns, size, format
rugo schema events.parquet            # column names, types, nullability
rugo preview -n 5 events.parquet       # first rows as a table
rugo describe events.parquet          # per-column null counts, min/max

# Reshape and move data between formats
rugo convert events.parquet events.jsonl   # format inferred from extensions
rugo merge part-*.parquet all.parquet      # concatenate schema-identical files
rugo split --rows 100000 big.parquet       # chunk into smaller files
rugo diff before.parquet after.parquet     # schema differences

# Every verb takes --json, so it composes with jq and shell pipelines
rugo count --json events.parquet | jq .num_rows

Each format has a notebook you can run in your browser, on the same 4,630 space launches: Parquet, CSV, JSONL and Avro (Google Colab).

A complete worked example on a real dataset is available in Google Colab or as a notebook on GitHub.

Parquet, CSV, JSONL — read and write. Avro — read.

Every format supports column projection; Parquet, CSV and JSONL also filter rows with predicates. Writers produce standard output readable by any compliant tool. Avro is reader-only — rugo doesn't write it.

from rugo import parquet

# Metadata only — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows)
print([c.name for c in meta.schema_columns])

# Selective streaming read
with parquet.read_parquet(
    "planets.parquet",
    columns=["id", "name"],
    predicates=[("id", ">", 4)],
) as reader:
    for morsel in reader:
        names = morsel.column("name").to_pylist()

# Write (ZSTD default)
data = parquet.write_parquet(morsel)
with open("out.parquet", "wb") as f:
    f.write(data)

# Stream batches at constant memory — one row group per call
with open("out.parquet", "wb") as f:
    with parquet.open_parquet_writer(f.write) as writer:
        for batch in batches:
            writer.write_row_group(batch)

from rugo import csv

# Schema without reading all data
meta = csv.read_metadata("data.csv")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with csv.read_csv(
    "data.csv",
    columns=["name", "score"],
    predicates=[("age", ">", 30)],
) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# TSV variant
with csv.read_csv("data.tsv", delimiter="\t") as reader:
    for morsel in reader: ...

# Write
csv_bytes = csv.write_csv(morsel)

from rugo import jsonl

# Schema without reading all data
meta = jsonl.read_metadata("events.jsonl")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with jsonl.read_jsonl(
    "events.jsonl",
    columns=["id", "status"],
    predicates=[("status", "==", "active")],
) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

# Write a Morsel to JSONL bytes
jsonl_bytes = jsonl.write_jsonl(morsel)

from rugo import avro

# Writer schema and codec, from the header — no record decoded
meta = avro.read_metadata("launches.avro")
print(meta.codec, meta.columns)

# Projected read — dotted names reach into nested records
with avro.read_avro(
    "launches.avro",
    columns=["Company", "Rocket.Name"],
) as reader:
    for morsel in reader:
        print(morsel.column("Rocket.Name").to_pylist())

# Read through a reader schema — drop, add (with defaults) or widen fields
schema = {"type": "record", "name": "Launch", "fields": [
    {"name": "Mission", "type": "string"},
    {"name": "Source", "type": "string", "default": "archive"},
]}
with avro.read_avro("launches.avro", reader_schema=schema) as reader:
    for morsel in reader: ...

Vectors and Morsels.

Rugo speaks Draken — a bundled columnar substrate that ships inside the wheel. There are two objects you will encounter.

vec = morsel.column("name")
values = vec.to_pylist()  # ["Mercury", "Venus", …]

for morsel in reader:
    print(len(morsel), "rows")
    vec = morsel.column("id")

A read → write round-trip works naturally: read_parquet yields Morsels, write_parquet / write_csv / write_jsonl all consume a Morsel.

from rugo import parquet
from rugo.csv import write_csv
from rugo.jsonl import write_jsonl

with parquet.read_parquet("planets.parquet") as reader:
    for morsel in reader:
        csv_bytes   = write_csv(morsel)
        jsonl_bytes = write_jsonl(morsel)
        pq_bytes    = parquet.write_parquet(morsel)

Rugo isn't designed to decode data faster.
It's designed to avoid decoding data you don't need.

Rugo was born out of Opteryx's need for a fast, low-memory Parquet reader — thin by design, and opinionated about what not to load.

Where to go next.