Convert NDJSON to CSV

The arithmetic here is simpler than for any other source on this site. NDJSON guarantees one complete record per line, so the number of rows you are about to get is the number of lines in the file — wc -l on the command line, or the line count in any editor that can open it. Nothing in the data can produce a line break, because a newline inside a string value is escaped inside the JSON.

That makes the conversion predictable in a way a JSON array is not. If the file has 412,000 lines you get 412,000 rows plus a header, and if the output has fewer, something was wrong with the input rather than with the conversion. Blank lines are skipped, including the trailing newline nearly every writer leaves at the end.

Each line in an NDJSON file is independent, and nothing obliges the second one to carry the keys of the first. A log stream that emits several event types — a request, an error, a job completion — puts all of them in one file with different fields each, which is perfectly reasonable as a stream and awkward as a table.

The conversion collects every key seen anywhere in the file and gives each one a column, leaving cells empty where a line had nothing. Nothing is lost, and the result can be very sparse: forty columns of which each row uses eight. If that describes your output, the useful move is to filter the file down to one event type before converting rather than to widen the spreadsheet — jq with a select on the event type takes seconds and produces a table you can actually read.

If a line is not valid JSON, the conversion does not continue past it. The message says which line: "This file could not be read as NDJSON — line 3 is not valid JSON." That is deliberate rather than fragile, and it is the behaviour worth wanting.

A broken line in a log almost never means a stray character. It means a truncated write — a process killed mid-flush, a rotated file cut at a boundary, a partial upload — and everything after that point is suspect. Skipping the line quietly would produce a table missing an unknown number of records with nothing to indicate it. With the line number in hand, deleting or repairing the tail takes one command, and you know exactly what you lost.

Structured logging usually nests: a request object with a method and a path, a user object with an id, a context block with a trace identifier. Those flatten into columns named for the path — request.method, user.id — with one column per leaf value.

One or two levels produce a table anybody can work with, and that covers most logging libraries’ defaults. Deeper structures widen quickly, and a log line carrying a whole serialised payload will produce a column per field inside it. When that happens the honest answer is that the interesting part of the file is one subtree, and extracting it before converting gives a far better table than flattening everything and hiding columns afterwards.

A record carrying a list — tags, error frames, a set of applied rules — is flattened into numbered columns: tags.0, tags.1, tags.2. Every value survives and the table gets a column for the longest list in the file, most of them empty.

For analysis that is rarely what you want. If the array is incidental, joining it into a single string before converting produces one readable column. If the array is the thing you are analysing — counting error frames, say — the shape you want is one row per element, which means expanding the file first. Both are a single jq expression, and both beat sorting a spreadsheet by tags.3.

Types are the main casualty, and it is worth being clear that they are lost at the destination rather than in transit. The values are written faithfully — a number as digits, a boolean as true or false, a null as an empty cell — but CSV has no type system, so whatever opens the file decides what everything is.

Excel decides badly and predictably: leading zeros vanish from identifiers, anything date-shaped becomes a date, and long numeric ids lose their last digits. The remedy is to import rather than open — Data, then From Text/CSV, with the identifier columns set to Text — or to convert to XLSX instead, where the types are declared in the file and nothing is guessed.

Values containing a comma, a quotation mark or a line break are wrapped in double quotation marks with internal quotes doubled, which is the RFC 4180 convention and is read correctly by every importer worth using. Everything else is written bare, so the file stays readable.

Log data leans on that mechanism more than most. User agents, request paths with query strings, error messages and stack traces are all full of commas and quotation marks. If the destination is a shell pipeline rather than a spreadsheet — awk, cut, a naive split on commas — the quoting is exactly what those tools do not implement, and converting to tab-separated values instead avoids the argument entirely.

The parse is fast and the ceiling is memory, because the whole file is read into records and the full set of columns worked out before the first row can be written. The free tier accepts up to 100 MB; tens of megabytes converts without drama and several hundred is where a browser tab starts to labour.

Above that, the source format is on your side. NDJSON splits safely at any line boundary — split -l 500000 produces valid files — so converting a very large log in pieces is legitimate rather than a workaround, and the pieces have identical headers if the event shapes are the same. For a file in the gigabytes, a streaming tool is the right instrument and this page is not.

The CSV is a view, not a replacement. It has lost the types, flattened the structure and reconciled records that were never obliged to agree, and none of that is recoverable from the table.

The original file is also better at the things you will want next. It appends without rewriting, it filters with jq without being loaded whole, and it can be converted again with a different subtree extracted when the question changes. Treat the CSV as the answer to today’s question and the NDJSON as the data.

The conversion is plain JavaScript running in this browser tab. No request carries the file anywhere, there is no account and no daily allowance, and you can confirm all of that by opening the network tab and converting something.

That is not a nicety for this pair. A production log holds IP addresses, session identifiers, request paths, user agents and often enough a query string with a token in it — the exact contents that make uploading one to a converter a reportable event in most organisations. Here there is nothing to upload.