Convert NDJSON to Parquet

This is what makes the pair interesting. Every line of an NDJSON file is independent, which is exactly why it is the format things append to: a new field can be added to the writer on a Tuesday and no existing line needs rewriting. Over a few months of deploys, a single file accumulates several generations of record shape.

Parquet has no tolerance for that. Its footer declares a fixed set of columns with one type each, and every row obeys it — which is what lets a query engine describe a two-gigabyte file without reading a row. So the conversion has to reconcile the generations into one schema, and everything below is a consequence of how it does that.

A field that first appears three hundred thousand lines into the file still becomes a column, and the rows before it get nulls. That is the only reconciliation that cannot lose data, and it is the reason the reader makes a full pass over the file before writing anything.

Most tools sample instead — the first thousand lines, the first megabyte — and it is why a load that worked on a test extract fails in production. The failure is worse than an error: a sampled schema quietly discards every field it did not see. Reading everything costs a pass over the data and removes that category of surprise entirely, at the price described in the section on memory below.

The other half of drift is a key that kept its name and changed its type. An identifier that used to be a number and is now quoted, a status that was a boolean and became a string, a version that went from 3 to "3.1" — all of them appear in long-lived event files, usually without anybody noticing at the time.

The type for each column is decided over every value in it: all booleans gives BOOLEAN, all whole numbers inside the 32-bit range gives INT32, other numbers give DOUBLE, and anything mixed gives a string column with the numeric values written as text alongside. A string column is visible, castable in one expression, and never silently drops a row — which the alternative, taking the first type and nulling out the rest, does. If a column you expected to be numeric arrives as text, querying the values that fail to cast is the fastest way to find the day the writer changed.

Structured events nest by convention: a request block, a user block, a context object carrying a trace identifier. Parquet has nowhere to put an object inside a cell, so each record is flattened to its leaves first and the path becomes the column name — request.method, user.id, context.trace_id. The types survive that, because they come from the JSON rather than from a re-parse of text.

Arrays are the case worth a look before you convert. They flatten by position, so a tags array of a and b becomes tags.0 and tags.1, and the column set is the union across every line in the file — one very long list somewhere in a month of events widens the schema for the whole month. Where a field is a genuinely variable-length list, joining it to a single string in jq first gives a schema you can query rather than one you have to explain.

Parquet has a 64-bit integer type and this conversion never emits it. Values pass through JavaScript numbers, which carry 53 bits of integer precision, so anything outside the 32-bit range is written as DOUBLE rather than claiming an exactness it no longer has.

Event files are full of the values this affects: millisecond epoch timestamps, snowflake-style identifiers, byte counters. For a timestamp, DOUBLE is fine — the precision is far beyond what a millisecond needs. For an identifier it is not, and the fix is at the writer that produced the events: emitting identifiers as JSON strings keeps them exact through every step of this pipeline and lands them as a string column, which is what an identifier should be.

Fifty thousand generated order records of six fields came to 207 KB as Parquet against 4.8 MB as NDJSON — better than twenty to one on this data. The shape of the saving matters more than the ratio, because the ratio depends entirely on the data.

Two things are happening. Every line of NDJSON repeats every key, so on a six-field record the field names are a substantial share of the bytes; in a column store each name is written once in the footer. And values of one kind sitting together compress far harder than the same values scattered through lines, which is why a city or event-type column costs almost nothing while a column of unique free text saves comparatively little.

There is a real irony here. NDJSON exists so that a consumer never has to hold the whole file, and this conversion holds the whole file — because the last line can add a column or change a column type, and the footer cannot be written until that is known.

So the ceiling is memory rather than a plan tier. The free tier accepts up to 100 MB; tens of megabytes converts without drama and several hundred is where a browser tab labours. Above that the source format is on your side: split -l 500000 produces valid files, each converts on its own, and a query engine reads a directory of Parquet files as one table. That is the arrangement those engines are built for, and it is a better answer than one very large file.

DuckDB reads the file in a FROM clause, pandas in one call, Spark as a native table. The first thing worth looking at is the column list and its types rather than the first ten rows, and there are three specific questions: which columns are mostly null, which arrived as strings that should be numbers, and how many columns there are at all.

Each answer is a fact about the source. Mostly-null columns mark where the record shape changed; string columns that should be numeric mark where a type changed under the same key; a column count larger than the field count marks a nested block or an array that flattened wider than you expected. Twenty seconds on the schema saves the hour it takes to notice any of them after the file has been joined to something else.

The natural unit for this conversion is a period rather than a whole history: a day of events, a month, one rotated log. Each converts to its own file, and DuckDB, Spark and every lakehouse table format read a directory of them as a single table with the schemas merged at read time.

That also handles drift better than one file does. When the record shape changes, the new files carry the new columns and the old ones do not, and the query engine reconciles them on read rather than the converter reconciling them on write. Partitioning by day and converting each day once is the shape this data wants, and it means never reconverting the history because one field was added.

The Parquet writer and the JSON parsing are both ordinary JavaScript, loaded on demand by this page — there is no WebAssembly module and no server step. No request carries the file anywhere; there is no account, no queue and no plan tier.

For telemetry that is usually the deciding factor. Event streams contain user identifiers, IP addresses, request paths and whatever else the application felt like recording, and the fact that this data is on its way into an analytics store does not make a third-party web converter an acceptable stop along the route. The registry also marks Parquet support as patchy for a reason: it is binary and not editable, so if the file is destined for a person rather than a query engine, a spreadsheet is the better target.