| Library | Installed | Cold import | Pulls in |
|---|---|---|---|
| rugo 0.5.5 | 9.4 MB | 14 ms | nothing |
| duckdb 2.0.0 | 51.7 MB | 69 ms | nothing |
| fastparquet 2026.9.0 | 74.3 MB | 605 ms | pandas, numpy, cramjam, fsspec |
| pyarrow 26.0.0 | 126.7 MB | 41 ms | nothing |
| polars 2.0.0 | 170.5 MB | 171 ms | polars-runtime-32 |
Peak memory on a selective read — 1% of rows, two columns, of a 509 MB, 100-column Parquet file. On one thread: rugo 78 MB, DuckDB 102 MB, PyArrow 127 MB, Polars 127 MB. On all cores: rugo 155 MB, DuckDB 185 MB, Polars 187 MB, PyArrow 247 MB.
Installed size
is everything pip install <package>
puts into site-packages, transitive
dependencies included, with a fresh virtualenv per
library so no library is measured in an environment
another has touched. macOS 26.6 on Apple silicon (arm64),
CPython 3.13, measured 2026-10-25. Cold import is the
wall-clock cost of importing each library's Parquet
entry point in a fresh interpreter, over and above bare
interpreter startup, best of 15 runs.
The wheel sizes shown on PyPI are compressed downloads
and are smaller than any of these numbers — rugo's
macOS arm64 wheel is 3.6 MB compressed and 9.4 MB
installed. Footprint also varies by platform: rugo's
Linux x86_64 wheel, the one AWS Lambda uses, is 14.0 MB
installed.
Peak memory is the process's peak resident set
(ru_maxrss), interpreter and library
import included, for a fresh process per library, on
the wide file described under Performance: 1,000,000
rows by 100 columns, zstd, 20 row groups, two columns
where an unsorted integer column keeps 1% of rows, so
no row group can be skipped. "On one thread" pins every
library to a single thread; "on all cores" is each
library's default. Results stay in each library's
native form and are not converted to Python lists.
Measured 2026-10-25 with rugo 0.5.5, DuckDB 2.0.0, Polars 2.0.0 and PyArrow 26.0.0
on an Apple M5 Pro, CPython 3.13. Reproduce it with
benchmarks/measure_parquet.py.
Traditional libraries optimise decoding speed. Rugo optimises reading less data in the first place — pushing column projection and predicate pushdown ahead of decode, so data you don't want is never touched. Extracted from years of work on the Opteryx SQL engine.
Often it comes from avoiding decoding data you don't need in the first place. Rugo was built as the file layer for the Opteryx SQL engine, where the goal was simple: if we don't need it, we don't read it. Column projection and row-group pruning happen before any decoding — so the columns and rows a query doesn't need are never paid for. The result is a library that gets faster the more selective your query is, stays tiny on disk, and carries no PyArrow, Pandas, or NumPy into your environment.
AWS Lambda and GCP Cloud Functions bill by memory and duration, and every cold start pays for the imports. Rugo's cold import is roughly 3× to 12× faster than PyArrow, DuckDB or Polars — 14 ms against 41, 69 and 171 — and it adds up fast at scale-out.
At 13× smaller than PyArrow and 18× smaller than
Polars, rugo shrinks Lambda deployment packages and
container images, speeds up deploys, and leaves
almost all of AWS Lambda's 250 MB unzipped limit
for your own code. The wheel bundles everything it
needs, so pip install rugo is the
entire dependency story.
Column projection and row-group pruning are first-class citizens. Skip the columns you don't need. Skip the row groups that can't match. The more selective the query, the bigger rugo's advantage — and Parquet reads peak at up to 1.9× less memory than PyArrow.
Being a query engine. Joins, SQL and aggregations belong to Opteryx, the engine built on rugo. Rugo stays the file layer, and wins where serverless and embedded workloads actually hurt: a package 13× smaller, a cold start nearly 3× faster, and up to 1.9× less peak memory than PyArrow.
Rugo wins by decoding less — up to 21× faster than PyArrow on selective JSONL reads. The more columns and rows a query lets rugo skip, the bigger the win, and even when it has to read everything it is still 2.8× faster. On Parquet, parallel row-group decode keeps it level with DuckDB, Polars and PyArrow.
Rugo's Parquet reader decodes row groups in parallel, so it keeps pace with DuckDB, Polars and PyArrow on the reads it was built for. Same file, same machine, every library in its default configuration.
Best of 7 runs, each library in a fresh process, with
the file already in the OS page cache. Rugo 0.5.5, DuckDB 2.0.0,
Polars 2.0.0 (scan_parquet) and PyArrow
26.0.0, CPython 3.13 on an Apple M5 Pro with 18 cores,
measured 2026-10-25.
Files. The wide file is 1,000,000 rows by
100 columns of integers, floats and strings, 509 MB, zstd,
20 row groups of 50,000 rows, written by PyArrow with no
bloom filters. The narrow file has the first four columns:
1,000,000 rows, 19 MB, also 20 row groups. The sort column
is id; the filter column k is
uniform and unsorted, so no row group can be skipped.
Filters on the sorted column, where rugo can prune row
groups, are in the results file alongside the rest.
Queries. "2 columns" selects
a and b; the filtered queries
keep ~10% and ~1% of rows by k;
"SELECT *" reads the narrow file in full; count(*) reads
only the footer's row count.
Threading and materialisation. DuckDB,
Polars and PyArrow use every core by default; rugo decodes
row groups across all cores by default too. The one-thread
chart pins every library to a single thread
(threads=1 in rugo, SET threads=1
in DuckDB, POLARS_MAX_THREADS=1,
use_threads=False in PyArrow). Results stay in
each library's native form (rugo Morsels, a Polars
DataFrame, Arrow tables) and are not converted to Python
lists.
Reproduce it with
benchmarks/measure_parquet.py;
the scripts it drives are
bench_parquet_one.py
and
gen_parquet.py.
This is the case that shows the architecture. PyArrow reads and decodes every column on every query regardless of projection or filter, so asking for one column costs it the same as asking for all 110.
Best of 5 runs, rugo 0.5.5 against PyArrow 26.0.0 (multithreaded), CPython 3.13 on an Apple-silicon Mac, measured 2026-10-25. Even the last row — no projection, no filter, nothing to skip — is a 2.8× win.
Against the two Python Avro readers: fastavro, and
avro, the Apache project's reference
implementation.
Rugo returns typed columns while the others return one Python dict per record, so rugo is also timed with every column converted to Python values: that's the like-for-like column if you want Python objects at the end.
Best of 5 runs, each reader on one thread, decoding the same
in-memory file: rugo 0.5.5, fastavro 1.13.1, avro 1.12.2,
CPython 3.13 on an Apple-silicon Mac, measured 2026-10-25.
That's 33× faster than fastavro and 208× faster than
avro reading every field, and still 7.3× and
46× once rugo's columns are turned into Python values.
fastavro and avro skip fields through a reader
schema; rugo is given columns. They are
smaller installs — 3.1 MB and 0.5 MB against rugo's
9.4 MB, which covers four formats. The file is written by
fastavro, with an event-stream mix of types: scalars,
strings, nullable unions, enums, timestamps, decimals and
nested records, arrays and maps. Reproduce it with
benchmarks/measure_avro.py.
All three write the same rows with the same schema.
Rugo encodes and compresses blocks on every core by
default; fastavro and avro run on one thread.
The one-thread column is rugo held to the same.
Best of 5 runs, each writer in a fresh process, writing to
an in-memory buffer: rugo 0.5.5, fastavro 1.13.1, avro 1.12.2,
CPython 3.13 on an Apple-silicon Mac (6 performance + 12
efficiency cores), measured 2026-10-25.
On one thread rugo is 4.7× faster than fastavro and 22×
faster than avro with deflate, 11× and 64×
with zstandard; on every core, 51× and 237× with deflate.
The files are about the same size — with deflate, rugo's is
23.4 MB, fastavro's 25.4 MB and avro's
23.8 MB. Rugo writes from columns (a Morsel); the others
write from one Python dict per record, their input, and
building either input is not timed. Each writer uses its
default block size. The rows are the reader benchmark's
data as rugo reads it — the nested record and map as JSON
text. Reproduce it with
benchmarks/measure_avro_write.py.
Every library writes the same table, from its own in-memory form.
Best of 5 runs, each library in a fresh process: rugo
0.5.5, PyArrow
26.0.0, polars 2.0.0, DuckDB 2.0.0 and
fastparquet 2026.9.0 (with pandas 3.1.0), CPython 3.13 on an
Apple-silicon Mac, measured 2026-10-25. The table is 200,000
rows by 50 columns of integers, floats and strings, loaded
into each library's own form before the clock starts. Every library runs at its
default threading (polars, DuckDB and PyArrow use every
core). Parquet is zstd for every writer. "Rugo" writes it
without its bloom filters and page index (both on by
default — with them it took ~70 ms), so its file holds
what a plain Parquet writer's does: statistics and
dictionaries included, as the other libraries write them at
their defaults. "Rugo (fast write)" is
write_parquet_fast: it also drops the
statistics and the dictionary. The file is valid Parquet
that any reader can read in full (33.6 MB against 38.4 MB),
but it carries nothing a reader could use to skip row
groups, so a selective scan reads all of it. Output goes to
memory, except DuckDB and fastparquet, which only write to a
file, so their times include that write. "Python" is the
standard library's json.dumps per row. Reproduce it with
benchmarks/measure_writes.py.
Rugo is the file layer for applications that read data selectively rather than scan it wholesale — places where footprint, cold-start time, and memory matter as much as throughput.
Rugo is the file layer under Opteryx, the SQL engine it was built for. Use it directly to scan Parquet with column projection and predicate pushdown and stream the result as Morsels, with no query engine in the way. When you want SQL, reach for Opteryx, which runs on rugo.
Thirty seconds, one install, no setup: read two columns of the successful launches in a Parquet file hosted on this site.
import urllib.request from rugo import parquet path, _ = urllib.request.urlretrieve("https://rugo.dev/data/space_missions.parquet") with parquet.read_parquet( path, columns=["Company", "Price"], predicates=[("Mission_Status", "==", "Success")], ) as reader: print(sum(len(morsel) for morsel in reader), "rows") # 4162 rows
Rugo reads and writes Parquet, CSV, JSONL and Avro. The API is the same shape across all four: pass a path or bytes, get columnar data back.
pip install rugo
from rugo import parquet # Read — context manager, yields Morsels with parquet.read_parquet("planets.parquet", columns=["id", "name"], predicates=[("id", ">", 4)]) as reader: for morsel in reader: print(morsel.column("name").to_pylist()) # Metadata — no column decode meta = parquet.read_metadata("planets.parquet") print(meta.num_rows, [c.name for c in meta.schema_columns]) # Write data = parquet.write_parquet(morsel, compression="zstd")
from rugo import csv # Same shape — context manager, yields Morsels with csv.read_csv("data.csv", columns=["name"], predicates=[("age", ">", 30)]) as reader: for morsel in reader: print(morsel.column("name").to_pylist()) meta = csv.read_metadata("data.csv") data = csv.write_csv(morsel)
from rugo import jsonl # Same shape again with jsonl.read_jsonl("events.jsonl", columns=["id", "status"], predicates=[("status", "==", "active")]) as reader: for morsel in reader: print(len(morsel), "rows matched") meta = jsonl.read_metadata("events.jsonl") data = jsonl.write_jsonl(morsel)
from rugo import avro # Same shape with avro.read_avro("events.avro", columns=["id", "device.os"]) as reader: for morsel in reader: print(morsel.column("device.os").to_pylist()) meta = avro.read_metadata("events.avro") # writer schema + codec data = avro.write_avro(morsel, codec="deflate")
The same install puts a rugo command on your
PATH — the reader and writer, driven from the
shell. Inspect, convert, and reshape files without writing
any Python.
# Look at a file rugo info events.parquet # rows, columns, size, format rugo schema events.parquet # column names, types, nullability rugo preview -n 5 events.parquet # first rows as a table rugo describe events.parquet # per-column null counts, min/max
# Reshape and move data between formats rugo convert events.parquet events.jsonl # format inferred from extensions rugo merge part-*.parquet all.parquet # concatenate schema-identical files rugo split --rows 100000 big.parquet # chunk into smaller files rugo diff before.parquet after.parquet # schema differences
# Every verb takes --json, so it composes with jq and shell pipelines rugo count --json events.parquet | jq .num_rows
Each format has a notebook you can run in your browser, on the same 4,630 space launches: Parquet, CSV, JSONL and Avro (Google Colab).
A complete worked example on a real dataset is available in Google Colab or as a notebook on GitHub.
Every format supports column projection; Parquet, CSV and JSONL also filter rows with predicates. Writers produce standard output readable by any compliant tool.
read_parquet(source, columns, predicates)returns context manager, one Morsel per row groupsource: str | bytescolumns: list[str] = Nonepredicates: list[tuple] = None(col, op, val) — ops: == != < <= > >= in "not in". Row-group pruning then row-level filter.read_metadata(source)returns ParquetMetadatasource: str | byteswrite_parquet(morsel, compression, bloom_filters, max_rows_per_row_group)returns bytescompression: str"zstd" or "none"bloom_filters: bool | list[str]max_rows_per_row_group: int0 for a single row groupopen_parquet_writer(sink, compression, bloom_filters)returns context manager, constant-memory streaming writesink: callablewriter.write_row_group(morsel)Decode: int32/64, float32/64, boolean, byte_array · UNCOMPRESSED, SNAPPY, ZSTD, GZIP, LZ4 · PLAIN, RLE_DICTIONARY, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY
Write: INT8–64, FLOAT32/64, BOOL, VARCHAR, DATE32, TIME, TIMESTAMP64, INTERVAL, DECIMAL, ARRAY — ZSTD or uncompressed, per-column min/max stats, optional bloom filters.
Rugo's Parquet advantage is reading less, not raw scan throughput. Row-group pruning and column projection cut what is decoded and keep peak memory low.
from rugo import parquet # Metadata only — no column decode meta = parquet.read_metadata("planets.parquet") print(meta.num_rows) print([c.name for c in meta.schema_columns]) # Selective streaming read with parquet.read_parquet( "planets.parquet", columns=["id", "name"], predicates=[("id", ">", 4)], ) as reader: for morsel in reader: names = morsel.column("name").to_pylist() # Write (ZSTD default) data = parquet.write_parquet(morsel) with open("out.parquet", "wb") as f: f.write(data) # Stream batches at constant memory — one row group per call with open("out.parquet", "wb") as f: with parquet.open_parquet_writer(f.write) as writer: for batch in batches: writer.write_row_group(batch)
read_csv(source, columns, predicates, delimiter, has_header)returns context manager, one Morselsource: str | bytescolumns: list[str] = Nonepredicates: list[tuple] = None(col, op, val) — ops: == != < <= > >=delimiter: str = ",""\t" for TSVhas_header: bool = Trueread_metadata(source)returns CsvMetadatasource: str | byteswrite_csv(morsel, delimiter, header)returns bytes (RFC 4180)delimiter: str = ","header: bool = True
Type inference: int64 → float64 → VARCHAR → null.
from rugo import csv # Schema without reading all data meta = csv.read_metadata("data.csv") print(meta.num_rows, [c["name"] for c in meta.schema_columns]) # Projected + filtered read with csv.read_csv( "data.csv", columns=["name", "score"], predicates=[("age", ">", 30)], ) as reader: for morsel in reader: print(morsel.column("name").to_pylist()) # TSV variant with csv.read_csv("data.tsv", delimiter="\t") as reader: for morsel in reader: ... # Write csv_bytes = csv.write_csv(morsel)
read_jsonl(source, columns, predicates, explicit_schema, infer_schema)returns context manager, one Morselsource: str | bytescolumns: list[str] = Nonepredicates: list[tuple] = None(col, op, val) — ops: == != < <= > >=explicit_schema: dict = Noneinfer_schema: bool = Trueread_metadata(source)returns JsonlMetadatasource: str | byteswrite_jsonl(morsel)returns bytes
Inferred types: int64, float64, bool, str, bytes, null, array[T].
JSONL timings are in Performance.
from rugo import jsonl # Schema without reading all data meta = jsonl.read_metadata("events.jsonl") print(meta.num_rows, [c["name"] for c in meta.schema_columns]) # Projected + filtered read with jsonl.read_jsonl( "events.jsonl", columns=["id", "status"], predicates=[("status", "==", "active")], ) as reader: for morsel in reader: print(len(morsel), "rows matched") # Write a Morsel to JSONL bytes jsonl_bytes = jsonl.write_jsonl(morsel)
read_avro(source, columns, reader_schema)returns context manager, Morsels of whole Avro blocks (up to 65,536 rows each)source: str | bytescolumns: list[str] = None"device.os") reads one field out of a nested record as a plain columnreader_schema: dict | str = Noneread_metadata(source)returns AvroMetadatasource: str | byteswrite_avro(morsel, codec, dictionary, max_threads)returns bytes, a standard Avro object container filecodec: str = "deflate"null, deflate, snappy, zstandard or bzip2dictionary: bool = Trueenum when that is smaller — rugo reads it back dictionary-encodedmax_threads: int = 0open_avro_writer(sink, codec)returns context manager; write_batch(morsel) per batchsink: callablef.write): the header, then each block as it's written — memory stays at one batch. write_avro_stream(morsels, sink) does the loop for you
Decode:
null, deflate, snappy, zstandard, bzip2 codecs · all primitive types, enum, fixed, date, time, timestamp-millis/-micros, decimal (up to 38 digits), uuid, arrays of scalars · records, maps and arrays of nested values arrive as JSON text · unions of ["null", T]
Encode:
integers (unsigned up to 32-bit), floats, booleans, strings, bytes, date, time and timestamp (to microseconds), decimal (as fixed), uuid (fixed(16)), arrays and nested arrays · a column with no nulls is written as plain T, one with nulls as ["null", T]
Avro carries no statistics, so
there's no predicates argument — filter
the morsels you get back. Avro timings are in
Performance.
from rugo import avro # Writer schema and codec, from the header — no record decoded meta = avro.read_metadata("launches.avro") print(meta.codec, meta.columns) # Projected read — dotted names reach into nested records with avro.read_avro( "launches.avro", columns=["Company", "Rocket.Name"], ) as reader: for morsel in reader: print(morsel.column("Rocket.Name").to_pylist()) # Read through a reader schema — drop, add (with defaults) or widen fields schema = {"type": "record", "name": "Launch", "fields": [ {"name": "Mission", "type": "string"}, {"name": "Source", "type": "string", "default": "archive"}, ]} with avro.read_avro("launches.avro", reader_schema=schema) as reader: for morsel in reader: ... # Write — one call, or streamed batch by batch data = avro.write_avro(morsel, codec="zstandard") with open("out.avro", "wb") as f, avro.open_avro_writer(f.write) as writer: for batch in batches: writer.write_batch(batch)
Rugo speaks Draken — a bundled columnar substrate that ships inside the wheel. There are two objects you will encounter.
A single typed column (similar to an Arrow Array).
to_pylist() → list · materialise as a Python listis_null() → Vector[bool] · null mask, one bool per rownull_bitmap() → bytes · raw 1-bit-per-row null bitmapin_list(values) → Vector[bool] · membership testbetween(lower, upper) → Vector[bool] · inclusive range testsum() / min() / max() → scalar · typed reductions; null-awaretake(indices) → Vector · gather rows by index listto_arrow() → pyarrow.Array · interop; requires PyArrow at runtimetype property · Draken type tag (e.g. DRAKEN_VARCHAR)len(vec) → int · row countvec = morsel.column("name") values = vec.to_pylist() # ["Mercury", "Venus", …]
A batch of rows across columns (similar to an Arrow RecordBatch).
column(name: str) → Vector · return a single column by namecolumn_names property → list[bytes] · ordered column namescolumn_types property → list · Draken type per columnschema property → list[dict] · name + type per columnselect(col_names: list) → Morsel · project to a subset of columnsrename(new_names: list) → Morsel · rename columns in orderfilter_mask(mask: Vector[bool]) → Morsel · keep rows where mask is truetake(indices) → Morsel · gather rows by index listslice(offset, length) → Morsel · contiguous row windowcombine(morsels) classmethod → Morsel · concatenate a list of Morselsfrom_vectors(col_names, col_vecs) classmethod → Morsel · construct from column name/vector pairsto_arrow() → pyarrow.Table · interop; requires PyArrow at runtimelen(morsel) → int · row countnum_columns property → int · column countnbytes property → int · approximate memory footprint in bytesfor morsel in reader: print(len(morsel), "rows") vec = morsel.column("id")
A read → write round-trip works naturally:
read_parquet yields Morsels,
write_parquet / write_csv /
write_jsonl all consume a Morsel.
from rugo import parquet from rugo.csv import write_csv from rugo.jsonl import write_jsonl with parquet.read_parquet("planets.parquet") as reader: for morsel in reader: csv_bytes = write_csv(morsel) jsonl_bytes = write_jsonl(morsel) pq_bytes = parquet.write_parquet(morsel)
Rugo isn't just designed to decode data faster.
It's designed to avoid decoding data you don't
need.
Rugo was born out of Opteryx's need for a fast, low-memory Parquet reader — thin by design, and opinionated about what not to load.
Complete API reference, all options, limitations, and design notes.
A complete workflow on a real dataset: schema inspection, filtered streaming, aggregation, and format conversion.
How to use rugo, one notebook per format: read a file, filter on read, load it into pandas, write and convert (the Avro notebook covers reading). Runs in your browser.
Rugo is developed inside opteryx-core.
Source and issue tracking are both there.
Eight short, runnable scripts: read, stream, convert and write Parquet, CSV, JSONL and Avro.
Pre-built wheels for Linux x86-64/aarch64, macOS arm64. Python 3.11+.