Reading a Parquet footer with rugo info
rugo inforugo schemarugo inspectrugo describe
Try it
curl -LO https://rugo.dev/data/space_missions.parquet
rugo info space_missions.parquet
Open a file in the browser viewer →
A Parquet file ends with a footer: the schema, the row groups, and for every column the null count and min/max values. Everything you want to know before you read a file is already in there, in the last couple of kilobytes. rugo info, schema, inspect and describe read that footer and nothing else, so you can check a file without paying to read it.
The examples use 4,630 orbital launches, from Sputnik onwards. Install rugo with pip install rugo, run the box above, and follow along.
The one-minute summary: rugo info
rugo info space_missions.parquet
path : space_missions.parquet
format : parquet
size_bytes : 93169
num_rows : 4630
num_columns : 8
Pass --json when a script is reading it:
rugo info --json space_missions.parquet
{
"path": "space_missions.parquet",
"format": "parquet",
"size_bytes": 93169,
"num_rows": 4630,
"num_columns": 8
}
The schema
rugo schema space_missions.parquet
name type nullable
-------------- ------------- --------
Company varchar True
Location varchar True
Price float64 True
Lauched_at timestamp[us] True
Rocket varchar True
Rocket_Status varchar True
Mission varchar True
Mission_Status varchar True
(Lauched_at is spelled that way in the source dataset. The footer records column names exactly as they were written.)
What the footer looks like on disk
The last eight bytes of every Parquet file are a 4-byte footer length followed by the magic bytes PAR1. A reader starts there and works backwards:
python3 -c "
import struct
data = open('space_missions.parquet', 'rb').read()
print('footer:', struct.unpack('<I', data[-8:-4])[0], 'bytes of', len(data))
print('magic :', data[-4:])
"
footer: 1688 bytes of 93169
magic : b'PAR1'
All of the metadata above fits in 1,688 bytes: 1.8% of the file.
Row groups and column statistics: rugo inspect
rugo inspect space_missions.parquet
path : space_missions.parquet
size_bytes : 93169
num_rows : 4630
num_row_groups : 1
schema:
name type nullable
-------------- ------------- --------
Company varchar True
Location varchar True
Price float64 True
Lauched_at timestamp[us] True
Rocket varchar True
Rocket_Status varchar True
Mission varchar True
Mission_Status varchar True
row group 0 (rows=4630):
name physical_type logical_type null_count distinct_count bloom min/max
-------------- ------------- ------------- ---------- -------------- ----- -------
Company byte_array varchar 0 False True
Location byte_array varchar 0 False True
Price float64 float64 3380 False True
Lauched_at int64 timestamp[us] 127 False True
Rocket byte_array varchar 0 False True
Rocket_Status byte_array varchar 0 False True
Mission byte_array varchar 0 False True
Mission_Status byte_array varchar 0 False True
Reading it:
- One row group, holding all 4,630 rows. Larger files have many, and each one carries its own statistics, which is what lets a reader skip row groups that can't match a filter.
physical_typevslogical_type: how the bytes are stored versus what they mean.Lauched_atis stored as anint64and meanstimestamp[us].null_count:Priceis empty on 3,380 of 4,630 rows (73%). You've learned that without reading a single value.min/max:Truemeans the footer holds a minimum and maximum for that column.
The statistics themselves: rugo describe
rugo describe space_missions.parquet
name type null_count min max distinct_count
-------------- ------------- ---------- ----------------------------------------------- -------------------------------------- --------------
Company varchar 0 AEB i-Space
Location varchar 0 Blue Origin Launch Site, West Texas, Texas, USA Xichang Satellite Launch Center, China
Price float64 3380 2.5 450.0
Lauched_at timestamp[us] 127 -386310720000000 1659101280000000
Rocket varchar 0 ASLV Zoljanah
Rocket_Status varchar 0 Active Retired
Mission varchar 0 -TJS_6.00 iPStar-1
Mission_Status varchar 0 Failure Success
Prices run from 2.5 to 450.0. Timestamp bounds are shown as raw microseconds since 1970-01-01, so the minimum is negative: it is the first launch in the file.
python3 -c "
from datetime import datetime, timedelta, timezone
print(datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(microseconds=-386310720000000))
"
1957-10-04 19:28:00+00:00
Sputnik 1, on the day it launched.
Proof that it never reads the data
Zero out most of the file, everything between the 4-byte header and the footer, and keep the footer intact:
python3 -c "
data = bytearray(open('space_missions.parquet', 'rb').read())
data[4:80000] = bytes(79996)
open('hollow.parquet', 'wb').write(data)
"
rugo info hollow.parquet
path : hollow.parquet
format : parquet
size_bytes : 93169
num_rows : 4630
num_columns : 8
The footer commands don't notice. Asking for the rows themselves does:
rugo preview hollow.parquet -n 2 >/dev/null 2>&1; echo "preview exit code: $?"
preview exit code: 1
When to reach for this
- Triage a pile of files. Row count, size and schema for each one, without paying to read any of them.
- Gate a pipeline.
rugo info --jsonand comparenum_rowsand the schema against what you expect before loading. - Plan a query.
null_countandmin/maxtell you which columns are worth reading and which row groups can be skipped.
Commands checked against rugo 0.5.12.