Reading Parquet in AWS Lambda without PyArrow

Justin Joyce · 17 October 2026 · 12 min read

aws lambda parquet serverless benchmarks

Correct at time of publishing. For up-to-date performance and feature information, see rugo.dev.

A Lambda deployment package can be 250 MB unzipped. PyArrow uses 69% of that on its own, Polars 79%, and neither can be uploaded directly to Lambda — both have to go via S3. rugo uses 5.4%. Those are measured numbers, and so is the more awkward one further down: on whole-file scans, DuckDB and Polars still read faster than rugo.

This is a walkthrough of putting a Parquet reader in a Lambda function — the deployment package, the handler code, and what it actually costs — comparing rugo, PyArrow, DuckDB and Polars. Everything here was built and deployed to AWS, not estimated.

The constraint

Lambda imposes three limits that matter here:

• 250 MB unzipped for the function code plus layers. This is the one that bites.
• 50 MB zipped for a direct upload through CreateFunction. Larger packages must be staged through S3.
• Cold starts. Every new execution environment pays to unpack your package and import your libraries before your handler runs. Lambda reports this separately as Init Duration.

A Parquet reader is rarely the only thing in the package. If the function also needs a model, an SDK, or a few application dependencies, the difference between spending 14 MB and 173 MB of the budget decides whether the thing deploys.

What each library costs

Each package was built exactly as a Lambda package is built — pip install <lib> --target ./package with Lambda's platform tags — then zipped. boto3 is provided by the runtime and is not bundled.

The last column is not a rounding judgement. Attempting a direct upload of the PyArrow or Polars package returns:

An error occurred (RequestEntityTooLargeException) when calling the
CreateFunction operation: Request must be smaller than 70167211 bytes
for the CreateFunction operation

A 55 MB zip looks like it should fit under a 70 MB cap — but the API takes the payload base64-encoded, which inflates it by a third to roughly 74 MB. Both packages have to be staged through S3 and deployed with --code S3Bucket=…,S3Key=…. PyArrow used to clear the bar: version 25.0.1's 48.8 MB zip fitted with about 7% to spare. Version 26.0.0 grew past it.

The deployment package

Build it on Linux, or with Linux platform tags, or the compiled wheels will be wrong for the runtime:

pip install rugo \
  --target ./package \
  --platform manylinux_2_34_x86_64 \
  --python-version 3.12 \
  --only-binary=:all: \
  --implementation cp

cp lambda_function.py ./package/
cd package && zip -r ../function.zip . && cd ..

aws lambda create-function \
  --function-name parquet-reader \
  --runtime python3.12 \
  --role arn:aws:iam::<account>:role/<your-lambda-role> \
  --handler lambda_function.handler \
  --zip-file fileb://function.zip \
  --timeout 120 --memory-size 1024 \
  --environment "Variables={BUCKET=your-bucket,KEY=events.parquet}"

manylinux_2_34_x86_64 matters. Lambda's python3.12 runtime is Amazon Linux 2023 (glibc 2.34), which is new enough for those wheels. Pinning the older manylinux2014 tag silently resolves some libraries to much older releases than you asked for — that happened while preparing this article, and produced a PyArrow 20 package when the intent was PyArrow 25.

The handler

rugo has no S3 filesystem — it reads a path or a bytes buffer, so you fetch the object yourself with boto3, which the runtime already provides. This handler reads two of eight columns, filters, and averages:

import os
import time

import boto3
from rugo import parquet

BUCKET = os.environ["BUCKET"]
KEY = os.environ["KEY"]
LOCAL = "/tmp/events.parquet"

_s3 = boto3.client("s3")


def handler(event, context):
    # /tmp survives between invocations on a warm environment, so a
    # cold start pays for this download and later invocations do not.
    if not os.path.exists(LOCAL):
        _s3.download_file(BUCKET, KEY, LOCAL)

    total = 0.0
    rows = 0
    with parquet.read_parquet(
        LOCAL,
        columns=["status", "response_ms"],
        predicates=[("status", "==", 200)],
    ) as reader:
        for morsel in reader:
            total += morsel.column("response_ms").sum()
            rows += len(morsel)

    return {"rows": rows, "mean_ms": round(total / rows, 4)}

That is the whole integration. pip install rugo pulls nothing else in, so there is no dependency resolution to manage and no transitive wheel to pin.

The benchmark

One file in S3: 38.6 MB of Parquet, 2 million rows, 8 columns, 20 row groups, ZSTD compressed — web-traffic shaped. Four Lambda functions, one per library, all python3.12 at 1024 MB in us-east-1, all doing the same work through the same shape of code.

Three query shapes, because one would not be honest:

• broad — status == 200, 2 of 8 columns. About 92% of rows pass, so there is almost nothing to skip. This shape measures raw decode.
• selective — status == 500, 1 of 8 columns. About 2% of rows pass, but status is randomly distributed, so every row group's min/max spans the predicate and no row group can be pruned. This isolates the benefit of column projection alone.
• pruned — a 50,000-row event_id window, 1 of 8 columns. event_id is monotonic across row groups, so 19 of 20 row groups are eliminated on footer statistics before anything is decoded.

The middle and last shapes are both "selective" in the sense of returning few rows, and they behave completely differently. Any Parquet benchmark that does not separate them is measuring luck.

All four libraries returned identical answers on all three shapes: 1,840,303 rows averaging 149.3266 ms, 40,193 rows averaging 149.3458 ms, and 50,000 rows averaging 149.3672 ms. Nothing below is a difference in what was computed.

Cold starts

Cold starts were forced by mutating each function's environment, which retires the execution environment, then reading Init Duration from the REPORT line. Init duration is the median of 20 cold starts per library; the rest is the median of five on this query.

rugo initialises fastest by a wide margin — 1.5× faster than DuckDB, 1.8× faster than PyArrow, 2.2× faster than Polars. Package size and import cost are the same story told twice.

The handler times include downloading the file from S3, about 600 ms for every library, so the read itself is the smaller part. DuckDB's handler finishes 19 ms sooner than rugo's, nowhere near enough to repay its 294 ms slower start. On first-request latency for this workload, rugo wins: 1,350 ms against DuckDB's 1,625 ms.

Read throughput, warm

With the file already on /tmp, the download drops out and what remains is the read:

rugo reads faster than PyArrow on the broad scan and level with it on the selective one. Row-group pruning does its job — 123 ms down to 6.4 ms, a 19× improvement once the predicate lines up with the data layout — and once the reader skips most of the file, all four finish within a millisecond of each other.

This is worth stating plainly, on rugo's own site: if your Lambda's job is to scan a lot of Parquet, rugo is not the fastest way to do it. DuckDB and Polars read the broad scan 20–35% faster. They have poured years into vectorised decode and it shows.

So when does rugo win?

The arithmetic is simple, and it is about the ratio of startup to work.

rugo starts roughly 300–650 ms sooner than the others. It then reads a full scan of this file 20–45 ms slower than DuckDB and Polars, and faster than PyArrow. If your function does one modest read per invocation, or starts cold often, rugo is ahead. If it stays warm and scans hundreds of megabytes per invocation, the deficit compounds and something else is the right answer. The follow-up works out where the break-even sits on your bill.

Concretely, rugo is the better choice when:

• The package budget is the binding constraint. If the function also carries an ML model or a pile of SDKs, 13.5 MB versus 173 MB is the difference between deploying and not. This is the strongest case by a distance.
• Invocations are short and frequent. High-concurrency, spiky traffic means many cold starts, and init cost is paid per environment while read cost is paid per request.
• Reads are small or well-pruned. Pulling a few thousand rows out of a partitioned lake, rather than aggregating everything.
• You want one dependency. No transitive wheels, no NumPy ABI to keep in step, no resolver surprises on a rebuild.

And it is the wrong choice when:

• You need throughput on large, warm scans. Use DuckDB — it reads the broad scan 20% faster than rugo here, and 61.7 MB is a sane package.
• You need SQL. rugo has none. DuckDB again.
• You need dataframe operations — joins, group-bys, window functions, reshaping. rugo has none of these. Use Polars, and stage the package through S3.
• You need the Arrow ecosystem. If something downstream wants pyarrow.Table, use PyArrow. rugo's to_arrow() requires PyArrow at runtime, which puts the 173 MB straight back.

What rugo cannot do

Beyond the capability gaps above, these are the concrete limits found while building this, all verified on rugo 0.5.4 against files written by PyArrow 26.0.0:

• No S3 or filesystem abstraction. There is no s3:// support; a URL string is treated as a filename and fails with RuntimeError: Unable to open parquet file. You fetch the bytes.
• No ranged reads. Because you supply the bytes, projection and pruning reduce decoding, not bytes transferred from S3. The whole object comes down regardless. PyArrow's and DuckDB's S3 integrations can fetch only the byte ranges they need; that is a real advantage on large files and rugo has no answer to it.
• Nested columns come back flattened. A struct column s is read as its leaf columns, s.x, s.y and so on; a map column comes back as two list columns, its keys and its values. The values are all there, but code expecting one column per top-level field has to reassemble them. list columns and decimal128 read as you'd expect.
• Brotli compression is not supported. Reading a Brotli-compressed column raises an error that names the codec. ZSTD, Snappy, GZIP, LZ4 and uncompressed columns all read correctly, as do INT96 and INT64 timestamps.

Reproducing this

Every number here came from a deployed function, not a local approximation. Sizes are from pip install --target trees built with Lambda's platform tags; cold starts are Init Duration from Lambda's own REPORT lines, median of 20 forced cold starts per library; read times are the median of ten warm invocations with the object already on /tmp. Python 3.12, x86_64, 1024 MB, us-east-1, measured 17 October 2026 against rugo 0.5.4, pyarrow 26.0.0, duckdb 1.5.6 and polars 2.0.0. The harness and raw results are in benchmarks/lambda-cost.

If your workload looks different from this one — and it probably does — the shape of the answer matters more than the numbers. Measure your own file, at your own memory setting, with your own query.