Validating JSONL line by line
rugo countrugo diff
Try it
curl -LO https://rugo.dev/data/space_missions.jsonl
rugo count space_missions.jsonl
JSON Lines is one JSON object per line, which makes it easy to write and easy to get subtly wrong: a crashed writer leaves a half-finished record, a stray log message lands in the middle, two producers disagree about a field. There is no footer to consult and no schema to check against. A JSONL file is valid when every line parses.
rugo parses every line whenever it reads a JSONL file, and when a line is bad it tells you which one. That makes rugo count a one-command validator. Install with pip install rugo and run the box above to get a sample file.
A clean file
rugo count space_missions.jsonl; echo "exit code: $?"
4630
exit code: 0
4,630 lines parsed, nothing wrong, exit code 0.
A broken file
Build a copy with a truncated record on line 4 and a stray line on line 8:
{
head -n 3 space_missions.jsonl
echo '{"Company": "RVSN USSR", "Price":'
sed -n '4,6p' space_missions.jsonl
echo 'not json'
sed -n '7p' space_missions.jsonl
} > broken.jsonl
rugo count broken.jsonl >/dev/null 2>&1; echo "exit code: $?"
exit code: 1
The exit code is what a pipeline needs. The message is what a person needs:
rugo count broken.jsonl 2>&1 | tail -n 1
ValueError: Malformed JSONL at line 4 (byte offset 696): '{"Company": "RVSN USSR", "Price":'
The line number is 1-based, and the byte offset is where that line starts (696 is the size of the three good lines before it), so you can jump straight to it with sed -n 4p broken.jsonl or an editor.
Finding every bad line
rugo stops at the first malformed line, which is what you want in a gate: fail early, say where. When you're cleaning a file up and need the whole list, a few lines of Python give you every line at once:
python3 -c "
import json, sys
for n, line in enumerate(open(sys.argv[1], 'rb'), start=1):
if line.strip():
try:
json.loads(line)
except ValueError:
print(f'line {n}: invalid JSON')
" broken.jsonl
line 4: invalid JSON
line 8: invalid JSON
Drop those lines (or send them to a dead-letter file), and the rest of the file is good:
sed '4d;8d' broken.jsonl > fixed.jsonl
rugo count fixed.jsonl
7
What rugo lets through
Blank lines are skipped and the last line doesn't need a trailing newline, so neither will trip your validation:
{ head -n 2 space_missions.jsonl; echo; sed -n 3p space_missions.jsonl; } > spaced.jsonl
rugo count spaced.jsonl
3
Catching schema drift
Valid JSON can still be the wrong shape. rugo diff compares the schemas of two files and exits 1 when they differ. Here a producer starts sending a new Crew field on one record:
sed '3s/^{/{"Crew": 3, /' space_missions.jsonl > drifted.jsonl
rugo diff space_missions.jsonl drifted.jsonl; echo "exit code: $?"
+ added: Crew
exit code: 1
One record out of 4,630 was enough to show up. Run it between yesterday's file and today's and you'll hear about added and removed columns the day they appear.
A validation gate for a folder
for f in *.jsonl; do
rugo count "$f" >/dev/null 2>&1 || echo "invalid: $f"
done
invalid: broken.jsonl
Swap the echo for exit 1 and it's a CI step.
Commands checked against rugo 0.5.12.