What is an NDJSON file?

NDJSON is a plain-text format you can open in any editor. It is used for moving data between programs and streaming.

The extension is .ndjson, and the full name is Newline-Delimited JSON. Both matter less than what the file can hold, which is what the rest of this page is about.

It dates from 2013.

Age is worth knowing here for one practical reason: the older a format is, the more programs have had time to learn it.

It is published in full, so anyone can implement it from the document rather than by inspection, which is why it turns up in so many programs and why files written twenty years ago still open. A published specification is not the same thing as a royalty-free one: where a format wraps a codec, the patent licensing is a separate question the standard does not settle.

NDJSON stores its content exactly. Saving it again changes nothing, so it can be opened, edited and re-saved as often as you like without accumulating damage — which is what makes it a working format rather than a delivery one.

NDJSON can hold a layout that lets it start playing before it has finished arriving.

That matters mostly when converting: whatever the target cannot hold is dropped, usually without a warning.

NDJSON has no comment syntax. Anything explanatory has to live outside the file, which is worth knowing before choosing it for something a human will edit by hand.

jq and pandas read it, and so do most programs of the same kind.

If a file will not open, the format is rarely the problem — it is more often that the program predates it. Converting to something older is the reliable way past that, and it is what the rest of this site is for.

No browser reads it.

That is the single most common reason to convert it: not that the format is bad, but that the place you want to show the file cannot read it.

NDJSON is meant to be opened and changed. Keep the file in this format for as long as the work is going on, and export from it whenever a finished copy is needed.

This is the problem the format exists for. A JSON document is only valid once the closing bracket arrives, so a parser handed a ten-gigabyte array of records has to read all ten gigabytes into memory before it can hand back the first one. For a log stream that never ends, it can never hand back anything at all.

NDJSON removes the array. Each line is a complete, independent JSON object, terminated by a newline and surrounded by nothing. A reader takes one line, parses it, processes it and forgets it — so memory use is the size of the largest record rather than the size of the file, and a process can start work on the first record immediately.

Adding a record to a JSON array means rewriting the file: the closing bracket is at the end, and something has to go before it. Adding a record to an NDJSON file means writing a line.

That is what makes it the natural format for anything accumulating over time — application logs, event streams, audit trails, scraped data, telemetry. It is also what makes it safe under concurrency in a way a JSON array is not: a single write of a line under the system’s atomic write size will not interleave with another writer’s line, so several processes can append to the same file without corrupting it.

A JSON array truncated by a crash or a full disk is invalid in its entirety — one missing bracket and a parser rejects the whole file, including the 99 per cent that arrived perfectly.

An NDJSON file truncated mid-write loses the last line and nothing else. Every complete line before it still parses, and a reader can skip the broken one and carry on. For data collected over weeks and stored on hardware that will eventually fail, that difference is not academic.

Log shipping and observability: Elasticsearch’s bulk API, Logstash, Fluentd, Vector and most structured logging libraries all speak it. Data exports from APIs that return more rows than fit in a response. Machine-learning datasets, where training data is one example per line and the file is read in a streaming loop.

Also as a wire format for long-running requests, where a server writes one JSON object per line as results become available and the client processes them as they arrive rather than waiting for the whole response.

One object per line, and no newlines inside it. A JSON string may legally contain an escaped newline and must never contain a literal one — a pretty-printed object spread over several lines breaks the format completely, and this is by far the commonest way an NDJSON file is generated wrongly.

UTF-8, no byte order mark, and a newline at the end of the last line rather than a missing one or a trailing blank. Line endings should be the single-character kind: a Windows carriage return before each newline is tolerated by most readers and rejected by some, which is exactly the sort of intermittent failure nobody enjoys diagnosing.

Three names for the same thing. NDJSON is the specification with a media type; JSON Lines is a separately written description of the identical format; JSONL is the extension people use, particularly in machine learning.

The differences between the specifications are cosmetic — a note about line endings here, a permitted extension there — and no tool in practice distinguishes them. A file with either extension can be handed to anything expecting the other.

To JSON, when a consumer wants a single document: wrap the lines in brackets and join them with commas. Trivial, and it reintroduces the memory problem, which is generally why the file was NDJSON to begin with.

To CSV, when the data is really flat records and somebody wants a spreadsheet. The catch is that JSON is nested and CSV is not, so nested objects have to be flattened into dotted column names and arrays have to be dropped or joined — a lossy step that is fine for reporting and wrong for an archive.

And to Parquet for anything analytical. A columnar, compressed, typed format reads dramatically faster for the queries people actually run against event data, and it is where a large NDJSON archive usually wants to end up.