Parquet, CSV and JSONL in 9.4 MB. No PyArrow.

Read and write Parquet, CSV, JSONL and Avro.
No PyArrow. No Pandas. No NumPy.

What it costs to read a Parquet file in Python

Library Installed Cold import Pulls in
rugo 0.5.5 9.4 MB 14 ms nothing
duckdb 2.0.0 51.7 MB 69 ms nothing
fastparquet 2026.9.0 74.3 MB 605 ms pandas, numpy, cramjam, fsspec
pyarrow 26.0.0 126.7 MB 41 ms nothing
polars 2.0.0 170.5 MB 171 ms polars-runtime-32

Peak memory on a selective read — 1% of rows, two columns, of a 509 MB, 100-column Parquet file. On one thread: rugo 78 MB, DuckDB 102 MB, PyArrow 127 MB, Polars 127 MB. On all cores: rugo 155 MB, DuckDB 185 MB, Polars 187 MB, PyArrow 247 MB.

How this was measured

Installed size is everything pip install <package> puts into site-packages, transitive dependencies included, with a fresh virtualenv per library so no library is measured in an environment another has touched. macOS 26.6 on Apple silicon (arm64), CPython 3.13, measured 2026-10-25. Cold import is the wall-clock cost of importing each library's Parquet entry point in a fresh interpreter, over and above bare interpreter startup, best of 15 runs.

The wheel sizes shown on PyPI are compressed downloads and are smaller than any of these numbers — rugo's macOS arm64 wheel is 3.6 MB compressed and 9.4 MB installed. Footprint also varies by platform: rugo's Linux x86_64 wheel, the one AWS Lambda uses, is 14.0 MB installed.

Peak memory is the process's peak resident set (ru_maxrss), interpreter and library import included, for a fresh process per library, on the wide file described under Performance: 1,000,000 rows by 100 columns, zstd, 20 row groups, two columns where an unsorted integer column keeps 1% of rows, so no row group can be skipped. "On one thread" pins every library to a single thread; "on all cores" is each library's default. Results stay in each library's native form and are not converted to Python lists. Measured 2026-10-25 with rugo 0.5.5, DuckDB 2.0.0, Polars 2.0.0 and PyArrow 26.0.0 on an Apple M5 Pro, CPython 3.13. Reproduce it with benchmarks/measure_parquet.py.

Traditional libraries optimise decoding speed. Rugo optimises reading less data in the first place — pushing column projection and predicate pushdown ahead of decode, so data you don't want is never touched. Extracted from years of work on the Opteryx SQL engine.

Performance doesn't only come from decoding faster.

Often it comes from avoiding decoding data you don't need in the first place. Rugo was built as the file layer for the Opteryx SQL engine, where the goal was simple: if we don't need it, we don't read it. Column projection and row-group pruning happen before any decoding — so the columns and rows a query doesn't need are never paid for. The result is a library that gets faster the more selective your query is, stays tiny on disk, and carries no PyArrow, Pandas, or NumPy into your environment.

⚡

Faster cold starts

AWS Lambda and GCP Cloud Functions bill by memory and duration, and every cold start pays for the imports. Rugo's cold import is roughly 3× to 12× faster than PyArrow, DuckDB or Polars — 14 ms against 41, 69 and 171 — and it adds up fast at scale-out.

📦

Smaller packages, no dependencies

At 13× smaller than PyArrow and 18× smaller than Polars, rugo shrinks Lambda deployment packages and container images, speeds up deploys, and leaves almost all of AWS Lambda's 250 MB unzipped limit for your own code. The wheel bundles everything it needs, so pip install rugo is the entire dependency story.

🔍

Lower memory, more selectivity

Column projection and row-group pruning are first-class citizens. Skip the columns you don't need. Skip the row groups that can't match. The more selective the query, the bigger rugo's advantage — and Parquet reads peak at up to 1.9× less memory than PyArrow.

What rugo doesn't chase

Being a query engine. Joins, SQL and aggregations belong to Opteryx, the engine built on rugo. Rugo stays the file layer, and wins where serverless and embedded workloads actually hurt: a package 13× smaller, a cold start nearly 3× faster, and up to 1.9× less peak memory than PyArrow.

The gap is projection, not decode speed.

Rugo wins by decoding less — up to 21× faster than PyArrow on selective JSONL reads. The more columns and rows a query lets rugo skip, the bigger the win, and even when it has to read everything it is still 2.8× faster. On Parquet, parallel row-group decode keeps it level with DuckDB, Polars and PyArrow.

Parquet — 1M rows, 100 cols, 509 MB

Rugo's Parquet reader decodes row groups in parallel, so it keeps pace with DuckDB, Polars and PyArrow on the reads it was built for. Same file, same machine, every library in its default configuration.

All cores

2.1×faster than DuckDB on a full read
3.9×faster than DuckDB on count(*)
1.2×faster than PyArrow, 2 of 100 columns
SELECT 2 columns
Rugo~4.0 ms
DuckDB~5.5 ms
polars~3.1 ms
PyArrow~4.7 ms
WHERE ~10% pass
Rugo~4.8 ms
DuckDB~5.5 ms
polars~2.9 ms
PyArrow~4.4 ms
WHERE ~1% pass
Rugo~4.2 ms
DuckDB~5.2 ms
polars~2.8 ms
PyArrow~4.2 ms
SELECT * — 4 columns
Rugo~2.9 ms
DuckDB~6.1 ms
polars~3.7 ms
PyArrow~4.3 ms
count(*)
Rugo~0.3 ms
DuckDB~1.3 ms
polars~0.6 ms
PyArrow~0.7 ms

One thread — a one-vCPU Lambda or container

1.4×faster than DuckDB on one thread
1.1×faster than PyArrow, full read, one thread
Paritywith polars on 2-column and full reads — within 2 ms
SELECT 2 columns
Rugo~22 ms
DuckDB~32 ms
polars~20 ms
PyArrow~26 ms
WHERE ~10% pass
Rugo~25 ms
DuckDB~29 ms
polars~21 ms
PyArrow~28 ms
WHERE ~1% pass
Rugo~25 ms
DuckDB~27 ms
polars~20 ms
PyArrow~27 ms
SELECT * — 4 columns
Rugo~30 ms
DuckDB~42 ms
polars~27 ms
PyArrow~33 ms
count(*)
Rugo~0.3 ms
DuckDB~1.2 ms
polars~0.5 ms
PyArrow~0.7 ms
How this was measured

Best of 7 runs, each library in a fresh process, with the file already in the OS page cache. Rugo 0.5.5, DuckDB 2.0.0, Polars 2.0.0 (scan_parquet) and PyArrow 26.0.0, CPython 3.13 on an Apple M5 Pro with 18 cores, measured 2026-10-25.

Files. The wide file is 1,000,000 rows by 100 columns of integers, floats and strings, 509 MB, zstd, 20 row groups of 50,000 rows, written by PyArrow with no bloom filters. The narrow file has the first four columns: 1,000,000 rows, 19 MB, also 20 row groups. The sort column is id; the filter column k is uniform and unsorted, so no row group can be skipped. Filters on the sorted column, where rugo can prune row groups, are in the results file alongside the rest.

Queries. "2 columns" selects a and b; the filtered queries keep ~10% and ~1% of rows by k; "SELECT *" reads the narrow file in full; count(*) reads only the footer's row count.

Threading and materialisation. DuckDB, Polars and PyArrow use every core by default; rugo decodes row groups across all cores by default too. The one-thread chart pins every library to a single thread (threads=1 in rugo, SET threads=1 in DuckDB, POLARS_MAX_THREADS=1, use_threads=False in PyArrow). Results stay in each library's native form (rugo Morsels, a Polars DataFrame, Arrow tables) and are not converted to Python lists.

Reproduce it with benchmarks/measure_parquet.py; the scripts it drives are bench_parquet_one.py and gen_parquet.py.

JSONL — 1M rows, 110 cols, 2.0 GB

This is the case that shows the architecture. PyArrow reads and decodes every column on every query regardless of projection or filter, so asking for one column costs it the same as asking for all 110.

21×faster than PyArrow, filtered to ~10%
16×faster, one column of 110
2.8×faster, every column
SELECT 1 column
Rugo~75 ms
PyArrow~1.22 s
WHERE ~10% pass
Rugo~74 ms
PyArrow~1.59 s
WHERE ~1% pass
Rugo~72 ms
PyArrow~1.58 s
SELECT *
Rugo~452 ms
PyArrow~1.29 s
How this was measured

Best of 5 runs, rugo 0.5.5 against PyArrow 26.0.0 (multithreaded), CPython 3.13 on an Apple-silicon Mac, measured 2026-10-25. Even the last row — no projection, no filter, nothing to skip — is a 2.8× win.

Avro — 200k rows, 20 fields, deflate, 21.6 MB

Against the two Python Avro readers: fastavro, and avro, the Apache project's reference implementation.

Why there are two rugo bars

Rugo returns typed columns while the others return one Python dict per record, so rugo is also timed with every column converted to Python values: that's the like-for-like column if you want Python objects at the end.

33×faster than fastavro
208×faster than avro (Apache)
Every field
Rugo~29 ms
Rugo → Python~130 ms
fastavro~953 ms
avro (Apache)~5.95 s
3 of 20 fields
Rugo~18 ms
Rugo → Python~24 ms
fastavro~750 ms
avro (Apache)~3.42 s
How this was measured

Best of 5 runs, each reader on one thread, decoding the same in-memory file: rugo 0.5.5, fastavro 1.13.1, avro 1.12.2, CPython 3.13 on an Apple-silicon Mac, measured 2026-10-25. That's 33× faster than fastavro and 208× faster than avro reading every field, and still 7.3× and 46× once rugo's columns are turned into Python values. fastavro and avro skip fields through a reader schema; rugo is given columns. They are smaller installs — 3.1 MB and 0.5 MB against rugo's 9.4 MB, which covers four formats. The file is written by fastavro, with an event-stream mix of types: scalars, strings, nullable unions, enums, timestamps, decimals and nested records, arrays and maps. Reproduce it with benchmarks/measure_avro.py.

Writing Avro — the same 200k rows, 20 fields

All three write the same rows with the same schema.

Why there are two rugo bars

Rugo encodes and compresses blocks on every core by default; fastavro and avro run on one thread. The one-thread column is rugo held to the same.

51×faster than fastavro
237×faster than avro (Apache)
4.7×faster than fastavro on one thread
deflate
Rugo~36 ms
Rugo, 1 thread~387 ms
fastavro~1.80 s
avro (Apache)~8.42 s
zstandard
Rugo~14 ms
Rugo, 1 thread~122 ms
fastavro~1.36 s
avro (Apache)~7.82 s
Uncompressed
Rugo~5.2 ms
Rugo, 1 thread~24 ms
fastavro~1.22 s
avro (Apache)~7.66 s
How this was measured

Best of 5 runs, each writer in a fresh process, writing to an in-memory buffer: rugo 0.5.5, fastavro 1.13.1, avro 1.12.2, CPython 3.13 on an Apple-silicon Mac (6 performance + 12 efficiency cores), measured 2026-10-25. On one thread rugo is 4.7× faster than fastavro and 22× faster than avro with deflate, 11× and 64× with zstandard; on every core, 51× and 237× with deflate. The files are about the same size — with deflate, rugo's is 23.4 MB, fastavro's 25.4 MB and avro's 23.8 MB. Rugo writes from columns (a Morsel); the others write from one Python dict per record, their input, and building either input is not timed. Each writer uses its default block size. The rows are the reader benchmark's data as rugo reads it — the nested record and map as JSON text. Reproduce it with benchmarks/measure_avro_write.py.

Writing Parquet and JSONL — 200k rows, 50 columns

Every library writes the same table, from its own in-memory form.

Parquet, zstd

5.5×faster than PyArrow
3.5×faster than DuckDB
5.7×faster than fastparquet
Rugo~66 ms
Rugo (fast write)~36 ms
polars~25 ms
DuckDB~229 ms
PyArrow~361 ms
fastparquet~372 ms

JSONL

29×faster than Python's json module
5.0×faster than DuckDB
Rugo~41 ms
polars~16 ms
DuckDB~208 ms
Python (json)~1.20 s
How this was measured

Best of 5 runs, each library in a fresh process: rugo 0.5.5, PyArrow 26.0.0, polars 2.0.0, DuckDB 2.0.0 and fastparquet 2026.9.0 (with pandas 3.1.0), CPython 3.13 on an Apple-silicon Mac, measured 2026-10-25. The table is 200,000 rows by 50 columns of integers, floats and strings, loaded into each library's own form before the clock starts. Every library runs at its default threading (polars, DuckDB and PyArrow use every core). Parquet is zstd for every writer. "Rugo" writes it without its bloom filters and page index (both on by default — with them it took ~70 ms), so its file holds what a plain Parquet writer's does: statistics and dictionaries included, as the other libraries write them at their defaults. "Rugo (fast write)" is write_parquet_fast: it also drops the statistics and the dictionary. The file is valid Parquet that any reader can read in full (33.6 MB against 38.4 MB), but it carries nothing a reader could use to skip row groups, so a selective scan reads all of it. Output goes to memory, except DuckDB and fastparquet, which only write to a file, so their times include that write. "Python" is the standard library's json.dumps per row. Reproduce it with benchmarks/measure_writes.py.

A different niche, not a smaller PyArrow.

Rugo is the file layer for applications that read data selectively rather than scan it wholesale — places where footprint, cold-start time, and memory matter as much as throughput.

🧮

SQL engines

📊

Embedded analytics

⚡

Serverless data processing

λ

AWS Lambda

☁️

Google Cloud Run

🤖

AI / RAG preprocessing

📦

Containers

⌨️

CLI tools

🪶

Low-memory data applications

Rugo is for reading, not loading.

Rugo is the file layer under Opteryx, the SQL engine it was built for. Use it directly to scan Parquet with column projection and predicate pushdown and stream the result as Morsels, with no query engine in the way. When you want SQL, reach for Opteryx, which runs on rugo.

🪶

Choose rugo when

  • package size, cold start or dependency count matters: Lambda, Cloud Run, containers, embedded
  • the workload is selective: a few columns, a filter, a slice of the file
  • you want a streaming Morsel interface and a small, steady memory footprint
  • you need file I/O, not a query engine
🦖

Choose Opteryx when

  • you want to write SQL over your Parquet, JSONL and CSV files
  • you need joins, aggregations and a query planner
  • you want projection pruning and predicate pushdown handled for you, on the same rugo readers

Meet Opteryx →

Try it on a public file

Thirty seconds, one install, no setup: read two columns of the successful launches in a Parquet file hosted on this site.

import urllib.request
from rugo import parquet

path, _ = urllib.request.urlretrieve("https://rugo.dev/data/space_missions.parquet")
with parquet.read_parquet(
    path,
    columns=["Company", "Price"],
    predicates=[("Mission_Status", "==", "Success")],
) as reader:
    print(sum(len(morsel) for morsel in reader), "rows")   # 4162 rows

One install, four formats.

Rugo reads and writes Parquet, CSV, JSONL and Avro. The API is the same shape across all four: pass a path or bytes, get columnar data back.

pip install rugo
from rugo import parquet

# Read — context manager, yields Morsels
with parquet.read_parquet("planets.parquet", columns=["id", "name"], predicates=[("id", ">", 4)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# Metadata — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows, [c.name for c in meta.schema_columns])

# Write
data = parquet.write_parquet(morsel, compression="zstd")
from rugo import csv

# Same shape — context manager, yields Morsels
with csv.read_csv("data.csv", columns=["name"], predicates=[("age", ">", 30)]) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

meta = csv.read_metadata("data.csv")
data = csv.write_csv(morsel)
from rugo import jsonl

# Same shape again
with jsonl.read_jsonl("events.jsonl", columns=["id", "status"], predicates=[("status", "==", "active")]) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

meta = jsonl.read_metadata("events.jsonl")
data = jsonl.write_jsonl(morsel)
from rugo import avro

# Same shape
with avro.read_avro("events.avro", columns=["id", "device.os"]) as reader:
    for morsel in reader:
        print(morsel.column("device.os").to_pylist())

meta = avro.read_metadata("events.avro")   # writer schema + codec
data = avro.write_avro(morsel, codec="deflate")

The same install puts a rugo command on your PATH — the reader and writer, driven from the shell. Inspect, convert, and reshape files without writing any Python.

# Look at a file
rugo info events.parquet              # rows, columns, size, format
rugo schema events.parquet            # column names, types, nullability
rugo preview -n 5 events.parquet       # first rows as a table
rugo describe events.parquet          # per-column null counts, min/max
# Reshape and move data between formats
rugo convert events.parquet events.jsonl   # format inferred from extensions
rugo merge part-*.parquet all.parquet      # concatenate schema-identical files
rugo split --rows 100000 big.parquet       # chunk into smaller files
rugo diff before.parquet after.parquet     # schema differences
# Every verb takes --json, so it composes with jq and shell pipelines
rugo count --json events.parquet | jq .num_rows

Each format has a notebook you can run in your browser, on the same 4,630 space launches: Parquet, CSV, JSONL and Avro (Google Colab).

A complete worked example on a real dataset is available in Google Colab or as a notebook on GitHub.

Parquet, CSV, JSONL and Avro — read and write.

Every format supports column projection; Parquet, CSV and JSONL also filter rows with predicates. Writers produce standard output readable by any compliant tool.

read_parquet(source, columns, predicates)returns context manager, one Morsel per row group
source: str | bytes
filename or buffer
columns: list[str] = None
columns to read; None = all
predicates: list[tuple] = None
(col, op, val) — ops: == != < <= > >= in "not in". Row-group pruning then row-level filter.
read_metadata(source)returns ParquetMetadata
source: str | bytes
reads footer only; no column decode
write_parquet(morsel, compression, bloom_filters, max_rows_per_row_group)returns bytes
compression: str
"zstd" or "none"
bloom_filters: bool | list[str]
True = all eligible cols; False = none; list = named cols only
max_rows_per_row_group: int
pass 0 for a single row group
open_parquet_writer(sink, compression, bloom_filters)returns context manager, constant-memory streaming write
sink: callable
called with each chunk of bytes as row groups are written
writer.write_row_group(morsel)
one parquet row group per call; same schema across calls

Decode: int32/64, float32/64, boolean, byte_array · UNCOMPRESSED, SNAPPY, ZSTD, GZIP, LZ4 · PLAIN, RLE_DICTIONARY, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY

Write: INT8–64, FLOAT32/64, BOOL, VARCHAR, DATE32, TIME, TIMESTAMP64, INTERVAL, DECIMAL, ARRAY — ZSTD or uncompressed, per-column min/max stats, optional bloom filters.

Where the Parquet advantage comes from

Rugo's Parquet advantage is reading less, not raw scan throughput. Row-group pruning and column projection cut what is decoded and keep peak memory low.

from rugo import parquet

# Metadata only — no column decode
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows)
print([c.name for c in meta.schema_columns])

# Selective streaming read
with parquet.read_parquet(
    "planets.parquet",
    columns=["id", "name"],
    predicates=[("id", ">", 4)],
) as reader:
    for morsel in reader:
        names = morsel.column("name").to_pylist()

# Write (ZSTD default)
data = parquet.write_parquet(morsel)
with open("out.parquet", "wb") as f:
    f.write(data)

# Stream batches at constant memory — one row group per call
with open("out.parquet", "wb") as f:
    with parquet.open_parquet_writer(f.write) as writer:
        for batch in batches:
            writer.write_row_group(batch)

Run the Parquet notebook in Colab → · download .ipynb

read_csv(source, columns, predicates, delimiter, has_header)returns context manager, one Morsel
source: str | bytes
filename or buffer
columns: list[str] = None
columns to read; None = all
predicates: list[tuple] = None
(col, op, val) — ops: == != < <= > >=
delimiter: str = ","
single character; use "\t" for TSV
has_header: bool = True
first row is column names
read_metadata(source)returns CsvMetadata
source: str | bytes
scans full file for row count; infers schema from sample
write_csv(morsel, delimiter, header)returns bytes (RFC 4180)
delimiter: str = ","
header: bool = True
include column name row

Type inference: int64 → float64 → VARCHAR → null.

from rugo import csv

# Schema without reading all data
meta = csv.read_metadata("data.csv")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with csv.read_csv(
    "data.csv",
    columns=["name", "score"],
    predicates=[("age", ">", 30)],
) as reader:
    for morsel in reader:
        print(morsel.column("name").to_pylist())

# TSV variant
with csv.read_csv("data.tsv", delimiter="\t") as reader:
    for morsel in reader: ...

# Write
csv_bytes = csv.write_csv(morsel)

Run the CSV notebook in Colab → · download .ipynb

read_jsonl(source, columns, predicates, explicit_schema, infer_schema)returns context manager, one Morsel
source: str | bytes
filename or buffer
columns: list[str] = None
columns to read; None = all
predicates: list[tuple] = None
(col, op, val) — ops: == != < <= > >=
explicit_schema: dict = None
override inferred types per column
infer_schema: bool = True
sample rows to determine types
read_metadata(source)returns JsonlMetadata
source: str | bytes
scans full file; infers schema from sample
write_jsonl(morsel)returns bytes

Inferred types: int64, float64, bool, str, bytes, null, array[T].

JSONL timings are in Performance.

from rugo import jsonl

# Schema without reading all data
meta = jsonl.read_metadata("events.jsonl")
print(meta.num_rows, [c["name"] for c in meta.schema_columns])

# Projected + filtered read
with jsonl.read_jsonl(
    "events.jsonl",
    columns=["id", "status"],
    predicates=[("status", "==", "active")],
) as reader:
    for morsel in reader:
        print(len(morsel), "rows matched")

# Write a Morsel to JSONL bytes
jsonl_bytes = jsonl.write_jsonl(morsel)

Run the JSONL notebook in Colab → · download .ipynb

read_avro(source, columns, reader_schema)returns context manager, Morsels of whole Avro blocks (up to 65,536 rows each)
source: str | bytes
filename or buffer — an Avro object container file
columns: list[str] = None
fields to read; None = all. A dotted name ("device.os") reads one field out of a nested record as a plain column
reader_schema: dict | str = None
the schema to read the file as: fields matched by name (or by field id, as Iceberg writes them), missing fields take their default, int→long→float→double widen
read_metadata(source)returns AvroMetadata
source: str | bytes
reads the header only: writer schema, codec, metadata map
write_avro(morsel, codec, dictionary, max_threads)returns bytes, a standard Avro object container file
codec: str = "deflate"
null, deflate, snappy, zstandard or bzip2
dictionary: bool = True
write a low-cardinality string column as an Avro enum when that is smaller — rugo reads it back dictionary-encoded
max_threads: int = 0
blocks are encoded and compressed on every core by default; the file is byte-identical at any thread count
open_avro_writer(sink, codec)returns context manager; write_batch(morsel) per batch
sink: callable
takes bytes (e.g. f.write): the header, then each block as it's written — memory stays at one batch. write_avro_stream(morsels, sink) does the loop for you

Decode: null, deflate, snappy, zstandard, bzip2 codecs · all primitive types, enum, fixed, date, time, timestamp-millis/-micros, decimal (up to 38 digits), uuid, arrays of scalars · records, maps and arrays of nested values arrive as JSON text · unions of ["null", T]

Encode: integers (unsigned up to 32-bit), floats, booleans, strings, bytes, date, time and timestamp (to microseconds), decimal (as fixed), uuid (fixed(16)), arrays and nested arrays · a column with no nulls is written as plain T, one with nulls as ["null", T]

No predicates on read

Avro carries no statistics, so there's no predicates argument — filter the morsels you get back. Avro timings are in Performance.

from rugo import avro

# Writer schema and codec, from the header — no record decoded
meta = avro.read_metadata("launches.avro")
print(meta.codec, meta.columns)

# Projected read — dotted names reach into nested records
with avro.read_avro(
    "launches.avro",
    columns=["Company", "Rocket.Name"],
) as reader:
    for morsel in reader:
        print(morsel.column("Rocket.Name").to_pylist())

# Read through a reader schema — drop, add (with defaults) or widen fields
schema = {"type": "record", "name": "Launch", "fields": [
    {"name": "Mission", "type": "string"},
    {"name": "Source", "type": "string", "default": "archive"},
]}
with avro.read_avro("launches.avro", reader_schema=schema) as reader:
    for morsel in reader: ...

# Write — one call, or streamed batch by batch
data = avro.write_avro(morsel, codec="zstandard")
with open("out.avro", "wb") as f, avro.open_avro_writer(f.write) as writer:
    for batch in batches:
        writer.write_batch(batch)

Run the Avro notebook in Colab → · download .ipynb

Vectors and Morsels.

Rugo speaks Draken — a bundled columnar substrate that ships inside the wheel. There are two objects you will encounter.

Vector

A single typed column (similar to an Arrow Array).

to_pylist() → list  ·  materialise as a Python list
is_null() → Vector[bool]  ·  null mask, one bool per row
null_bitmap() → bytes  ·  raw 1-bit-per-row null bitmap
in_list(values) → Vector[bool]  ·  membership test
between(lower, upper) → Vector[bool]  ·  inclusive range test
sum() / min() / max() → scalar  ·  typed reductions; null-aware
take(indices) → Vector  ·  gather rows by index list
to_arrow() → pyarrow.Array  ·  interop; requires PyArrow at runtime
type property  ·  Draken type tag (e.g. DRAKEN_VARCHAR)
len(vec) → int  ·  row count
vec = morsel.column("name")
values = vec.to_pylist()  # ["Mercury", "Venus", …]

Morsel

A batch of rows across columns (similar to an Arrow RecordBatch).

column(name: str) → Vector  ·  return a single column by name
column_names property → list[bytes]  ·  ordered column names
column_types property → list  ·  Draken type per column
schema property → list[dict]  ·  name + type per column
select(col_names: list) → Morsel  ·  project to a subset of columns
rename(new_names: list) → Morsel  ·  rename columns in order
filter_mask(mask: Vector[bool]) → Morsel  ·  keep rows where mask is true
take(indices) → Morsel  ·  gather rows by index list
slice(offset, length) → Morsel  ·  contiguous row window
combine(morsels) classmethod → Morsel  ·  concatenate a list of Morsels
from_vectors(col_names, col_vecs) classmethod → Morsel  ·  construct from column name/vector pairs
to_arrow() → pyarrow.Table  ·  interop; requires PyArrow at runtime
len(morsel) → int  ·  row count
num_columns property → int  ·  column count
nbytes property → int  ·  approximate memory footprint in bytes
for morsel in reader:
    print(len(morsel), "rows")
    vec = morsel.column("id")

A read → write round-trip works naturally: read_parquet yields Morsels, write_parquet / write_csv / write_jsonl all consume a Morsel.

from rugo import parquet
from rugo.csv import write_csv
from rugo.jsonl import write_jsonl

with parquet.read_parquet("planets.parquet") as reader:
    for morsel in reader:
        csv_bytes   = write_csv(morsel)
        jsonl_bytes = write_jsonl(morsel)
        pq_bytes    = parquet.write_parquet(morsel)

Rugo isn't just designed to decode data faster.
It's designed to avoid decoding data you don't need.

Rugo was born out of Opteryx's need for a fast, low-memory Parquet reader — thin by design, and opinionated about what not to load.

Where to go next.

📖

Full documentation

Complete API reference, all options, limitations, and design notes.

🚀

Example notebook

A complete workflow on a real dataset: schema inspection, filtered streaming, aggregation, and format conversion.

Run in Colab → View on GitHub
🧪

Try each format

How to use rugo, one notebook per format: read a file, filter on read, load it into pandas, write and convert (the Avro notebook covers reading). Runs in your browser.

Parquet → CSV → JSONL → Avro →
💻

Source

Rugo is developed inside opteryx-core. Source and issue tracking are both there.

📂

Examples

Eight short, runnable scripts: read, stream, convert and write Parquet, CSV, JSONL and Avro.

📦

PyPI

Pre-built wheels for Linux x86-64/aarch64, macOS arm64. Python 3.11+.