Read and write Parquet, CSV, and JSONL.
No PyArrow. No Pandas. No NumPy.

Traditional libraries optimise decoding speed. Rugo optimises reading less data in the first place — pushing column projection and predicate pushdown ahead of decode, so data you don't want is never touched. Extracted from years of work on the Opteryx SQL engine.

Performance doesn't only come from decoding faster.

Often it comes from avoiding decoding data you don't need in the first place. Rugo was built as the file layer for the Opteryx SQL engine, where the goals were simple: read as little data as possible, hold as little in memory as possible. Column projection and row-group pruning happen before any decoding — so the columns and rows a query doesn't need are never paid for. The result is a library that gets faster the more selective your query is, stays tiny on disk, and carries no PyArrow, Pandas, or NumPy into your environment.

Where PyArrow is faster: full-table scans with no filtering. PyArrow's Parquet engine has had over a decade of optimisation poured into raw decode throughput — Rugo doesn't compete there. It competes on how little it has to decode in the first place.

The gap is projection, not decode speed.

Rugo does not decode faster than PyArrow. It decodes less. That means the numbers below are not a single speedup figure — they move with how selective the read is, and on an unselective read they converge.

JSONL — 1M rows, 110 cols, 2.7 GB

This is the case that shows the architecture. PyArrow reads and decodes every column on every query regardless of projection or filter, so asking for one column costs it the same as asking for all 110.

Best of 3 runs, PyArrow multithreaded. Note the last row: with no projection and no filter the advantage nearly vanishes, which is the honest shape of this design.

CSV — wide file, 50 cols, 200k rows, 67 MB

CSV is the weaker case, and the table shows it. Rugo is at parity on a full scan and at parity on projection; the gap only opens once a predicate eliminates rows before the typed column build.

On narrow CSV files PyArrow is faster across the board. The crossover is driven by how many columns can be skipped and how many rows are eliminated before the typed column build — a narrow file gives Rugo nothing to skip.

A different niche, not a smaller PyArrow.

Rugo is the file layer for applications that read data selectively rather than scan it wholesale — places where footprint, cold-start time, and memory matter as much as throughput.

One install, three formats.

Rugo reads and writes Parquet, CSV, and JSONL. The API is the same shape across all three: pass a path or bytes, get columnar data back.

from rugo import parquet

# Read — context manager, yields Morsels
with parquet.read_parquet("planets.parquet", columns=["id", "name"], predicates=[("id", ">", 4)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# Metadata — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows, [c.name for c in meta.schema_columns])

# Write
data = parquet.write_parquet(morsel, compression="zstd")

from rugo import csv

# Same shape — context manager, yields Morsels
with csv.read_csv("data.csv", columns=["name"], predicates=[("age", ">", 30)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

meta = csv.read_metadata("data.csv")
data = csv.write_csv(morsel)

from rugo import jsonl

# Same shape again
with jsonl.read_jsonl("events.jsonl", columns=["id", "status"], predicates=[("status", "==", "active")]) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

meta = jsonl.read_metadata("events.jsonl")
data = jsonl.write_jsonl(morsel)

# Look at a file
rugo info events.parquet              # rows, columns, size, format
rugo schema events.parquet            # column names, types, nullability
rugo preview -n 5 events.parquet       # first rows as a table
rugo describe events.parquet          # per-column null counts, min/max

# Reshape and move data between formats
rugo convert events.parquet events.jsonl   # format inferred from extensions
rugo merge part-*.parquet all.parquet      # concatenate schema-identical files
rugo split --rows 100000 big.parquet       # chunk into smaller files
rugo diff before.parquet after.parquet     # schema differences

# Every verb takes --json, so it composes with jq and shell pipelines
rugo count --json events.parquet | jq .num_rows

A complete worked example on a real dataset is available in Google Colab or as a notebook on GitHub.

Parquet, CSV, JSONL — read and write.

All three formats support column projection and predicate filtering. Writers produce standard output readable by any compliant tool.

from rugo import parquet

# Metadata only — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows)
print([c.name for c in meta.schema_columns])

# Selective streaming read
with parquet.read_parquet(
    "planets.parquet",
    columns=["id", "name"],
    predicates=[("id", ">", 4)],
) as reader:
    for morsel in reader:
        names = morsel.column("name").to_pylist()

# Write (ZSTD default)
data = parquet.write_parquet(morsel)
with open("out.parquet", "wb") as f:
    f.write(data)

# Stream batches at constant memory — one row group per call
with open("out.parquet", "wb") as f:
    with parquet.open_parquet_writer(f.write) as writer:
        for batch in batches:
            writer.write_row_group(batch)

from rugo import csv

# Schema without reading all data
meta = csv.read_metadata("data.csv")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with csv.read_csv(
    "data.csv",
    columns=["name", "score"],
    predicates=[("age", ">", 30)],
) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# TSV variant
with csv.read_csv("data.tsv", delimiter="\t") as reader:
    for morsel in reader: ...

# Write
csv_bytes = csv.write_csv(morsel)

from rugo import jsonl

# Schema without reading all data
meta = jsonl.read_metadata("events.jsonl")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with jsonl.read_jsonl(
    "events.jsonl",
    columns=["id", "status"],
    predicates=[("status", "==", "active")],
) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

# Write a Morsel to JSONL bytes
jsonl_bytes = jsonl.write_jsonl(morsel)

Vectors and Morsels.

Rugo speaks Draken — a bundled columnar substrate that ships inside the wheel. There are two objects you will encounter.

vec = morsel.column("name")
values = vec.to_pylist()  # ["Mercury", "Venus", …]

for morsel in reader:
    print(len(morsel), "rows")
    vec = morsel.column("id")

A read → write round-trip works naturally: read_parquet yields Morsels, write_parquet / write_csv / write_jsonl all consume a Morsel.

from rugo import parquet
from rugo.csv import write_csv
from rugo.jsonl import write_jsonl

with parquet.read_parquet("planets.parquet") as reader:
    for morsel in reader:
        csv_bytes   = write_csv(morsel)
        jsonl_bytes = write_jsonl(morsel)
        pq_bytes    = parquet.write_parquet(morsel)

Rugo isn't designed to decode data faster.
It's designed to avoid decoding data you don't need.

Rugo was born out of Opteryx's need for a fast, low-memory Parquet reader — thin by design, and opinionated about what not to load.

Where to go next.