Convert Parquet to NDJSON

A Parquet file declares the type of every column in its footer. Convert it to CSV or to tab-separated text and that declaration is gone: every value becomes characters, and whatever reads the file next infers types again from what the characters look like — usually differently, and always without the benefit of the schema that was sitting right there.

JSON has types of its own. A number stays a number, a boolean stays a boolean, and a null stays null, so the record that arrives at the other end says the same thing about itself that the Parquet column did. That is not a small difference when the destination is a service or a queue with a schema, because it removes an entire class of "the API rejected it because the field was a string" from the middle of the pipeline.

This is the limitation worth knowing before you build anything on the output. Parquet holds lists, structs and maps natively, and those columns are written into the JSON as strings containing JSON rather than as nested JSON. A list column arrives as `"[\"a\",\"b\"]"`, which is a correct and complete record of the value and is not the shape a consumer expecting an array will accept.

It is one pass to fix. `jq -c '.tags |= fromjson'` over the converted file turns that field into a real array, and the same applies to a struct column. Doing it explicitly is also a chance to decide what the field should actually be called and whether the destination wants it nested at all — many will happily take the string. What matters is knowing, because a field that looks nested in a code review and is a string at runtime is an unpleasant thing to discover in an integration test.

JSON has no date type. A timestamp column is therefore written as an ISO 8601 string — `2024-03-11T09:30:00.000Z` — which every language, database and schema validator parses without being told a format, and which sorts chronologically as plain text.

The alternative would be an epoch number, and it is worse in a record that a person may have to read: a field holding 1710149400000 is unintelligible without knowing both the epoch and the unit, and the two common units differ by a factor of a thousand. If the destination insists on epoch milliseconds, converting from ISO is one expression and the direction is unambiguous; going the other way from a bare number requires an assumption you would have to document.

Parquet has a 64-bit integer type and JSON numbers are, in practice, limited by what the parsers on either end can represent exactly — which for most of them is 53 bits. A value beyond that cannot be carried as a number without changing it.

The conversion writes those as strings. A value inside the safe range stays a number, one outside it becomes a quoted string with every digit intact, and a column that spans the boundary will contain both. That mixed type is mildly annoying and it is the correct trade: an order reference that has been rounded looks exactly like an order reference, matches nothing, and is very hard to trace. If the destination has a schema, declare that field as a string and cast on arrival.

The file is one complete JSON object per line and nothing else — no opening bracket, no commas between records, no closing bracket. A parser expecting a JSON array will fail on the second line, and that is the format working as intended rather than a fault.

What the absence buys is that a consumer can read a line, act on it, discard it and move on, whatever the size of the file. It also means the output is addressable by line: `head -n 1000` gives a sample that is itself valid input to everything else, `split -l` produces loadable chunks, and a failed replay can resume from a line number rather than starting again.

This is the common shape of the job. A table sits in a data platform, and something outside it — a search index, a message queue, a service being backfilled, a staging environment that needs realistic data — takes JSON and has never heard of Parquet. NDJSON is the format that sits between them, and it needs no framing beyond a newline.

A shell loop reading lines and posting each one is enough for a small backfill, and for a larger one the same file feeds a bulk endpoint or a producer script without modification. The one thing to check first is field names: a data platform gives columns names shaped by the pipeline that produced them, and a service usually wants something else. Renaming with `jq` before sending is a single expression and is easier to review than a mapping buried in the consumer.

A null in a Parquet column is written as `null` in the JSON, which preserves a distinction that no delimited format can carry: null and the empty string are different values, and they arrive different.

That matters where the destination has a schema. A field declared as a string will reject null unless it is declared nullable, and a validator will tell you so at the record it happened on rather than at load time. Both of those are better failures than the delimited alternative, where a null and an empty string are the same zero characters and the difference has already been lost by the time anything checks it.

The output is much larger than the file it came from. Every record repeats every field name, every string is quoted, and none of the compression, dictionary encoding or binary representation that made the Parquet file small survives into text.

That is acceptable because this file is a transport artefact. It exists to be read once by a consumer and then deleted, and the durable copy stays as Parquet. Where the NDJSON is being kept rather than consumed, that is a signal that something has gone wrong in the design — a compressed columnar file is better at every part of storing data than a text file with the field names repeated on every row.

The Parquet reader is a library loaded on demand by this page and the JSON is written there, so no request carries the file. For an extract taken from the middle of a platform — before the aggregation, before the masking — that is normally the constraint that decides whether an online converter can be used at all.

Every row is materialised before anything is written, so memory is the ceiling. Parquet is compressed, so a modest file can expand a great deal and the limit arrives sooner than the size on disk suggests. For a large table, DuckDB will read the Parquet file and write newline-delimited JSON in one statement without holding either, and it lets you select only the columns the destination actually wants.

If the consumer can read Parquet, let it. Every warehouse, every query engine and most modern data libraries do, and passing the original preserves the schema, keeps the transfer small and removes a step in which the types can drift.

Convert when the consumer genuinely speaks JSON and nothing else, which is most services, most queues and most search indexes. That is a real and permanent divide — operational systems exchange records, analytical systems exchange columns — and this conversion is the bridge across it rather than a substitute for either side.