Analysing 1 GB of JSONL logs in Python: grep vs pandas vs rugo

Justin Joyce · 10 July 2026 · 8 min read

jsonl logs pandas performance

Same question, same file, same answer: 899,041 matching rows. pandas needed 2,500 MB of RAM to produce it. rugo needed 5 MB — 500× less — and finished 11× sooner. If you have ever had a log-analysis script die in a 512 MB container, that ratio is the whole story.

We needed rolling 1-hour average response times for the /api/data endpoint, computed from 1 GB of JSONL traffic logs, on an M3 MacBook. Three approaches, all returning identical results:

The row count is the part that makes the rest of the table worth reading. It is easy to look fast by doing less work, or by quietly dropping rows on a parse error. All three approaches agreed on the count, and agreed on the overall average response time to within 0.01 ms. The speed and memory differences are not a difference in what was computed.

The problem

The logs are JSONL — one JSON object per line. Each line looks like this:

{"timestamp": "2026-06-15T14:32:11", "url": "/api/data", "method": "GET",
 "status": 200, "response_ms": 142.3, "user_agent": "Mozilla/...",
 "bytes_sent": 12450}

After 30 days a single file is over 1 GB, roughly 6 million entries. We want rolling 1-hour average response times for /api/data — which tells us whether latency is trending up, not merely what the average was overall. About 15% of traffic hits that endpoint, and we need 3 of the 8 fields.

Those two ratios — 3 of 8 columns, 15% of rows — are what the rest of this article is about.

Approach 1: grep + Python

grep is the obvious first move for searching logs:

import subprocess
import json

result = subprocess.run(
    ["grep", "/api/data", "logs.jsonl"],
    capture_output=True, text=True,
)

entries = []
for line in result.stdout.split("\n"):
    if line:
        entry = json.loads(line)
        if entry["url"] == "/api/data":
            entries.append(entry)

The split between the two halves matters. grep does its filtering in about 0.4 seconds — it is genuinely excellent at text matching. But every matched line then has to be parsed into a Python dictionary, and that costs roughly 6.5 seconds.

7 seconds total, and that is before you write a single line of the rolling-average calculation. Memory stays reasonable at ~42 MB because the work is streamed, but you are hand-rolling the statistics yourself.

Approach 2: pandas

pandas is the standard answer for time-series analysis:

import pandas as pd

df = pd.read_json("logs.jsonl", lines=True)
df = df[df["url"] == "/api/data"]
df = df.sort_values("timestamp")

df["ts"] = pd.to_datetime(df["timestamp"])
df = df.set_index("ts")
df["rolling_avg"] = df["response_ms"].rolling("1h").mean()

It works, and it is by far the most expressive of the three. But read_json parses every column of every row, and the entire 1 GB file is materialised in memory before the first filter runs.

13.4 seconds, and ~2,500 MB peak RSS. The filter on the second line discards 85% of what the first line just spent two and a half gigabytes building. This is the reason log-processing scripts fall over in serverless functions and small containers: not that pandas is slow, but that its memory ceiling is set by the size of your input file rather than the size of your answer.

Approach 3: rugo

rugo is built around a different principle: read less data in the first place.

from rugo import jsonl

timestamps = []
values = []

with jsonl.read_jsonl(
    "logs.jsonl",
    columns=["url", "response_ms", "timestamp"],
    predicates=[("url", "==", "/api/data")],
) as reader:
    for morsel in reader:
        timestamps.extend(morsel.column("timestamp").to_pylist())
        values.extend(morsel.column("response_ms").to_pylist())

Two things happen before any data becomes a Python object:

1. Column projection — only 3 of the 8 fields are read. method, status, user_agent and bytes_sent are never parsed.
2. Predicate pushdown — only rows where url == "/api/data" are parsed. At ~15% of traffic, 85% of the file never becomes an object at all.

Results arrive as Morsels — batches of rows streamed from the file — so memory is bounded by the batch, not the file. Peak RSS was ~5 MB.

1.23 seconds. Same answer.

The numbers

The file: 1.05 GB, 6 million log entries, 15% matching /api/data. All three agree on 899,041 rows and on the overall average to within 0.01 ms.

Why rugo is faster here

It is not that rugo parses JSON faster than pandas. It is that rugo parses less JSON.

When the file has 8 columns and you need 3, pandas still reads and parses all 8. When 15% of rows match your filter, pandas loads 100% of them first and discards the rest afterwards. rugo pushes both decisions ahead of the decode, so the skipped columns and skipped rows are never paid for in either time or memory.

The corollary is the honest limit of the approach: the advantage scales with selectivity. Ask for every column of every row and there is nothing to skip, and the gap collapses. This design wins on selective reads and merely ties on unselective ones.

When to use what

grep when you need lines matching a pattern and you are going to read them as text. Nothing beats it at that. Just be aware that the moment you add Python JSON parsing behind it, the pipeline is slower than rugo.

pandas when you genuinely need the whole frame — all columns, no filtering, many operations per row — and you have the memory. Its ecosystem and expressiveness are real advantages that neither of the other two match.

rugo when you need specific columns from specific rows of a large file, or when your memory ceiling is fixed and your input size is not.

Reproducing this

pip install rugo is the whole install. There are no runtime dependencies — no PyArrow, no pandas, no NumPy — and the installed footprint is 11.6 MB with a 30 ms cold import (linux x86_64, CPython 3.12; see the footprint comparison for how that was measured and how it compares).

The benchmark script is available if you want to run these numbers yourself. rugo reads and writes Parquet, CSV and JSONL through the same API shape.

These logs come from opteryx.app, and the original write-up — with more on why the endpoint was being monitored — is on the Opteryx engineering blog as Monitoring opteryx.app — A Faster Way to Search Web Traffic Logs.