Convert JSON to Parquet

A Parquet file carries a schema in its footer: every column has a name and exactly one type, and every row obeys it. That is what lets a query engine tell you the shape of a two-gigabyte file without reading a row of it, and it is the property the format exists for.

A JSON array offers no such guarantee. The records are objects, each free to carry whatever keys it has, and nothing in the format prevents the fortieth record from holding a string in a field the first thirty-nine held a number. The conversion has to manufacture the schema the file never had, and every awkward case on this page comes from that one sentence.

Parquet has no place to put an object inside a cell, so the record is flattened before it is transposed: a customer object holding a city becomes a column named customer.city, carrying the type it had in the JSON. This is the same flattening the CSV, TSV, SQL and XLSX writers on this site do, deliberately so — one nested extract gives the same column names whichever of the five you send it to.

Arrays are the case to think about, because they flatten by position rather than by name. A tags array of a and b becomes two columns, tags.0 and tags.1, and the column list is the union across the whole file — so a field that is usually two items and occasionally forty produces forty columns, thirty-eight of them null in almost every row. Where a list is genuinely variable-length, joining it to a single string in jq before converting gives a schema you can query.

Where every record is a flat object — an event log, an extract of orders, a table an API returned one page at a time — the conversion does exactly what you want and does it in one step. The keys become columns in the order they were first seen, the values become typed column data, and the result is a file DuckDB will query without any preparation.

That covers a large share of real JSON extracts. Analytics events, billing lines, form submissions and most paginated list endpoints are flat by design, because the systems producing them are writing rows. A record that does nest still converts — it arrives wider, with a dotted column per leaf — so the glance at the first record is about knowing what the schema will look like rather than about whether it will work.

Records that disagree about which keys they have are reconciled by taking all of them. A field present in the first record and absent from the next produces a column with a null in the second row, and a field that only appears in the last ten records of a million still becomes a column.

That reconciliation reads the whole array before writing anything, which is the honest cost of not sampling. The benefit is that a rare field is never silently dropped, and the surprise it produces is predictable: converting a ten-record sample gives fewer columns than converting the whole file. Take the schema from a conversion of the whole extract, never from the sample you tested with.

JSON has real types, which is a genuine advantage over converting from a spreadsheet or a CSV: a number arrives as a number and a boolean as a boolean, with nothing inferred from the characters. Booleans become BOOLEAN, whole numbers within the 32-bit range become INT32, anything else numeric becomes DOUBLE, and text stays text.

The collision case is where records disagree. A field that is a number in most records and a string in a few — an identifier some systems quote and others do not, a quantity with "n/a" in it — makes the entire column text, and the numeric values are written as text alongside. That is deliberate: taking the first record type and nulling out the rest produces a file that loads cleanly and has quietly deleted the values that did not fit. A string column is visible and can be cast in one expression.

Parquet has a 64-bit integer type and this conversion does not emit it. The values pass through JavaScript numbers, which carry 53 bits of integer precision, so a value beyond the 32-bit range is written as DOUBLE instead of claiming an exactness it no longer has.

For identifiers that matters, and the fix is upstream. A JSON export that writes order references as strings keeps them exact through this conversion and lands them as a text column, which is what an identifier should be anyway. An export that writes them as bare numbers has already lost precision past sixteen digits before any converter sees the file.

Fifty thousand generated order records of six fields — an id, a product code, a city, a quantity, a price and a flag — came to 207 KB as Parquet. The same records were 4.8 MB as compact JSON and 6.8 MB pretty-printed, which is the form most exports actually arrive in.

The gap is not compression alone. Every JSON record repeats every key, so the field names appear fifty thousand times each; in a column store they appear once, in the footer. Repeated values compress hard when they sit together, which is why the city and product code columns cost almost nothing here. Expect a large saving on operational data of this kind and a much smaller one on records that are mostly distinct free text.

A JSON extract is often an object with one key holding the array — data, results, records. That single-key envelope is unwrapped for you and the array inside is what gets converted, because it is the shape almost every paginated endpoint returns and unwrapping it is what was meant.

Two keys is where it stops. A file shaped {"meta": {...}, "data": [...]} has no obvious array to prefer, so the whole object becomes one row and the records are flattened by index into it: data.0.id, data.1.id, on for as many records as the file holds. The symptom is unmistakable once you know it — a Parquet file with one row and several thousand columns. Cut the file down to the array with jq before converting.

DuckDB reads the file directly in a FROM clause, pandas in a single call, and Spark treats it as a native table. The file begins and ends with the four bytes PAR1, which is how any of them recognise it.

The first thing to look at is the schema rather than the first ten rows. Two questions answer almost everything: is any column typed as a string that you expected to be numeric, and is the column count the one you expected. The first tells you which records disagreed about a type; the second tells you whether a nested field or a variable-length array widened the file more than you meant it to. Both are far cheaper to discover now than after the file has been joined to three others.

In the browser. The Parquet writer is plain JavaScript that this page loads on demand — no WebAssembly and no server — so no request carries the extract anywhere and there is no account, queue or plan tier. The free tier accepts up to 100 MB.

The real ceiling is memory rather than that number, because the whole array is parsed, transposed into columns and written, so the entire dataset exists at once. Tens of megabytes is routine and several hundred is where a tab begins to labour. Past that point a streaming reader in a script is the right instrument, and saying so is better than failing halfway through a large file. The registry also notes Parquet support as patchy for a reason: it is binary, it is not editable, and a colleague without the right tooling cannot open it at all — if the file is going to a person rather than an engine, send a spreadsheet.