Convert TOML to NDJSON

A TOML document is a table. There is no such thing as a TOML file that is a list at the top level — the specification does not allow it — so the record boundary that newline-delimited JSON needs never appears inside a single file. The whole document becomes one line.

For most sources that would be a limitation. Here it is the shape of the problem: nobody wants a pyproject.toml split into fragments, and everybody who converts one to NDJSON is building a collection where each file is a record. The unit of the format and the unit of the question happen to agree, which is not true of every pair on this site.

The workflow is: convert each file, append the line to a growing output. Because every line is a complete, self-contained JSON value and nothing wraps the file — no enclosing array, no commas between records — concatenation is the entire merge step. Two lines from two conversions are a valid two-record NDJSON file.

What that gets you is a dataset you can ask questions of. Which crates pin a specific dependency version, which packages declare no licence, which repositories still target an old edition — all of them are one query over the collection instead of a script that walks directories and parses files. Building that inventory is much more useful than reading any single config, and it is the reason this pair exists.

A record with no source is close to useless in an inventory. The converter reads the contents of a file and nothing else — it does not know the path, the repository or the commit — so the line it produces has no field saying where it came from.

Add it yourself, at one of two moments. A jq pass during concatenation can inject a path key per line, which keeps the TOML files untouched and is the right approach for a fleet you do not own. Alternatively, a key inside each TOML file names it, which survives every future conversion but means editing files that belong to other people. The first scales; the second is more honest when the file really should identify itself.

Compact JSON: no indentation, no spaces after colons, keys in the order the TOML declared them, ending with a newline. Tables become nested objects and arrays of tables become arrays of objects, so an authors list or a set of build targets arrives in the shape a query engine expects.

The line is as long as the config is. A pyproject.toml is a few kilobytes and a generated lock-adjacent config can be far larger, and a reader that processes a line at a time has to hold the whole line in memory. That is rarely a problem at config sizes, and it is worth knowing before somebody points this at a file that is not really a config.

TOML has real temporal types and JSON does not, so every date and datetime becomes a string. Offsets survive rather than being normalised to UTC, a local date arrives exactly as written, and a local time gains a millisecond component it did not have in the source. Most loaders will read those as text unless the target schema says otherwise.

Two float literals are lossy in a way nothing reports: inf and nan are valid TOML and become null in JSON, so a configured infinite limit and an unconfigured one become indistinguishable in the dataset. The integer-float distinction goes as well, since 1.0 serialises as 1 — which matters if a query is meant to tell a version from a count.

TOML 1.0 requires 64-bit signed integers and JSON numbers are IEEE doubles, so a value above roughly nine quadrillion cannot be represented exactly. The parser stops with an error naming the line rather than writing a rounded number into your dataset.

That is the correct behaviour for an inventory in particular, where a wrong number would be indistinguishable from a right one across ten thousand records. When it happens, quote the value as a string in the source file — it was an identifier rather than a quantity in every real case where this comes up.

Configs from different projects will not have the same sections. One declares a tool table with three linters in it, the next declares none, a third uses a key nobody else uses. Every line carries only the keys its file had, which is a real advantage over flattening the same collection into a table where the columns would have to be the union of everything.

The destination decides how much that costs. A schema-on-read store handles ragged records natively and lets a query ask for a key that only some records carry. A loader with a fixed schema will reject the outliers or drop the fields. Sample the widest and the narrowest files in the collection before defining the target, rather than after loading eight thousand records.

TOML comments are the reasoning in a config: why a dependency is pinned, which ticket a workaround belongs to, what a magic number means. NDJSON is JSON per line and JSON has no comment syntax, so all of it is gone from the dataset.

That is acceptable for an inventory and not acceptable as a migration. Nobody querying which packages pin a version needs the surrounding prose; anybody replacing the TOML files with something generated from this data would be throwing it away. Use the stream to answer questions about the configs, and leave the configs where they are.

If you only have one config to read, convert it to JSON instead. Indented JSON is readable, jq handles it identically, and a one-line file is worse for every purpose except appending. NDJSON earns its place at the point where there are many files and they are going into something.

The other boundary is recurrence. If the inventory has to be rebuilt on a schedule, the work belongs in a script that walks the tree, parses each file with a real TOML library and emits the lines with the path already attached. This converter is for building the dataset the first time, deciding whether the questions it answers are worth automating, and doing it without any of the files leaving your machine.