Convert NDJSON to TOML

People assume TOML cannot hold what JSON holds, and for a list of records that is wrong. An array of tables is written as the same header repeated in double brackets — [[items]] three times gives three records — with each block carrying its own keys underneath. It is the construct Cargo uses for dependencies and Hugo uses for menu entries.

Compared with the JSON it came from it is more verbose and considerably easier to edit. Adding a record means copying four lines rather than balancing brackets and remembering a comma, and a diff of a change to one record touches only that block. For a file a person maintains, that is the entire argument.

A TOML document has to be a table at the top level, and a newline-delimited file is a list with no name of its own. So the list gets a placeholder name: the output is a run of [[items]] blocks, one per line of the source.

That name is the first thing to change, and it is a search and replace over the block headers — to [[dependencies]], [[authors]], [[products]], whatever the tool reading the file expects. There is nothing in an NDJSON file that could tell the converter what the records are, so the placeholder is deliberate and deliberately obvious rather than a guess that might look plausible and be wrong.

This is the quiet advantage of TOML over every tabular target for the same file. A CSV, a Parquet file or a run of INSERT statements has to reconcile records into one set of columns, filling in blanks where a record had nothing. An array of tables does not: each block carries exactly the keys its line had.

So a file where later records gained a field converts with no nulls, no empty cells and no widening. The result reads as what it is — a list of records that are mostly alike — rather than as a sparse grid. For a fixture set assembled over time, that is usually the more honest representation.

TOML has no null literal, and the writer does not invent one. A key whose JSON value was null is simply not written into the block, at the top level of a record or anywhere inside it.

Whether that matters is a question about the tool reading the file. Many treat a missing key and an empty value identically, falling back to a default either way, and for those nothing was lost. Where the distinction drives behaviour, the conversion has quietly changed the data, and the cheapest defence is to search the source for null before converting rather than to compare the two files afterwards — an absent line is much harder to spot than a changed one.

A record with a nested object produces a sub-table under its block: an author object inside a record becomes an [items.author] header with its keys beneath. Depth is expressed in the header rather than in indentation, so even three levels stay flat on the page.

It is legal at any depth and unpleasant past two or three. A header like [items.metadata.source.system] is valid and nobody enjoys reading it, and a fixture file that produces those is usually carrying more structure than a hand-edited file should. That is a signal about the data rather than about the conversion.

A TOML table header claims every line written after it until the next header, so a plain key placed below a block would silently become part of that block. The writer therefore puts any top-level scalar above the first header.

For this pair that rarely comes up, because the input is a list and everything ends up inside a block. It matters when you edit the file afterwards: adding a version key at the bottom of the file puts it inside the last record rather than at the top level, which is the single most common mistake people make with TOML. Add such keys at the top, above the first double-bracket header.

TOML 1.0 has four date and time types, including an offset date-time that is a real value rather than text. JSON has none — every timestamp in every JSON file is a string or a number by convention — so a value of "2024-01-01T00:00:00Z" is written as a quoted TOML string.

That is correct and it is not what the format could express. If the tool reading the file wants a real date-time, removing the quotation marks on those lines is the whole fix, and nothing else in the file has to change. It is a reasonable thing to do while you are in there renaming the block headers.

A few dozen is comfortable. A few hundred is a file nobody will read but a tool will still parse quickly. A few thousand blocks is a configuration file in name only, and at that point every property that made TOML attractive — legible diffs, hand editing, comments next to values — has stopped applying.

The stopping rule is worth applying honestly: if nobody is going to open the file, do not convert it to a configuration format. CSV is smaller and every tool reads it, JSON is what the data already was, and a database is what you want if the list is going to be queried. TOML earns its place only when a person is going to maintain the result.

Comments, which is the whole reason config formats look like this. JSON has none, and neither does a line-delimited file made of it, so the reasons behind the values in a fixture set currently live in a commit message or in nobody’s head at all.

Once the file is TOML, those reasons can sit next to the entries: why this record is excluded from a test, which upstream identifier a value has to match, what breaks if an entry is reordered. That is the information a generated file could never carry, and the first minutes after conversion are the cheapest time to write it down.

Both halves are JavaScript in this tab: the file is parsed line by line and the TOML is written by a small library loaded on demand. Nothing is uploaded, there is no account, and the free tier accepts up to 100 MB — which for a file this conversion is appropriate for is not a limit anyone approaches.

The privacy point still applies even at this scale. A fixture set is often real data with the names left in, and a seed file for a development database is frequently a slice of production. Not sending it anywhere is a smaller decision here than on a log file, and it is still the right default.