Convert YAML to NDJSON

NDJSON is JSON with a rule about newlines: one complete value per line, no commas, no enclosing brackets. To produce it, something has to decide where one record ends and the next begins, and the only thing in a YAML file that says so is a sequence at the top level. A document that begins with a list of dashes converts into one line per dash.

Everything else is one line. A mapping at the root is a single value no matter how much is nested inside it, so a Kubernetes manifest, a compose file or an application config comes out as one line of several kilobytes. That is not a failure and it is usually not what the person converting wanted, which is why it is the first thing on this page rather than a footnote.

If the records you want are nested under a key — a users key holding a list, an events key holding a list — the conversion will not reach in and find them. It looks at the root and nothing else. The fix is to lift that list to the top of the document before converting, which in an editor is deleting the first line and outdenting the rest.

The alternative is to convert to JSON instead and use jq to both select and stream: extracting the array and emitting it one element per line is a single expression. That is the better route when the extraction is going to be repeated, because it is scriptable, and this converter is the better route when it happens once.

YAML anchors let one block be defined once and referenced many times, and the parser resolves every alias into a full copy before anything is written. The NDJSON is therefore larger than the YAML, sometimes considerably.

For a stream that is not a cost, it is the requirement. A consumer of NDJSON reads one line at a time and may never have seen the others — that is the entire point of the format, and it is what lets a loader split a file across workers or resume after a failure. A line that referred to a definition on an earlier line would break under exactly the conditions the format exists to survive. Anything that reduces the duplication has to happen after the load, not in the file.

A YAML stream separated by --- markers is, conceptually, already NDJSON: one document per record, in order. It is the closest structural match in this whole family of formats, and it is the one input this conversion rejects, because the parser reads a single document and stops when it finds a second.

The workaround is a text edit rather than a tool. Replace each --- with a top-level list item marker and indent the documents beneath it, and the file converts into exactly the stream you expected. On a large file, a sed one-liner does it; on a handful of documents, so does an editor. It is worth saying plainly that this is the pipeline missing a feature rather than a limitation of either format.

NDJSON has been the interchange format for line-oriented tooling since around 2013, and the destinations all want slightly different things. BigQuery and Snowflake take newline-delimited JSON directly as a load format. jq reads it without any flag and writes it back with the compact option. pandas reads it with a lines argument. Any of those consume this output unchanged.

Elasticsearch is the exception people trip over. Its bulk API is NDJSON-shaped but expects an action line before every document, so a file of plain records is rejected. Interleaving the action lines is one jq or awk pass over the output, and it is deliberately not done here, because the action line names an index and an operation that only you know.

Each line is compact JSON: no indentation, no spaces after colons, keys in the order the YAML had them. The file is UTF-8 and ends with a newline, which most loaders require and some silently tolerate.

Types come from the YAML parser, not from the writer. Under YAML 1.2 an unquoted date stays a string, so a date field arrives as text rather than as anything a loader will recognise as a timestamp without a schema hint. Large integers are the one silent hazard: an identifier written 9223372036854775807 comes out as 9223372036854776000, because JSON numbers are IEEE doubles. Quote those in the YAML before converting, and they arrive as strings, which is what an identifier should have been anyway.

A list whose items do not share the same keys converts without complaint, and every line carries only the keys its record had. That is a real advantage over converting the same data to CSV, where the columns have to be the union of every record and the missing cells have to be filled with something.

Whether the destination is as relaxed is another question. A schema-on-read store takes ragged lines happily; a table loader with a fixed schema will reject or null out the unexpected fields. The conversion preserves what the YAML said, and it is worth scanning the first and last few lines of a large output to see how much the shape drifts across the file.

YAML comments are dropped, and NDJSON has nowhere to put them: it is JSON per line, and JSON has no comment syntax. There is also no header line, no schema declaration and no metadata block, so the file carries records and nothing else.

That absence is deliberate in the format and worth respecting in the pipeline. If the load needs to know a source, a batch identifier or a schema version, that belongs in a field on every record or in the name of the file, not in a preamble — anything at the top of an NDJSON file is a record, and a loader will try to read it as one.

If the YAML is configuration rather than data, this conversion has nothing to offer. A single line of minified JSON is worse than the YAML in every way a person cares about, and no stream consumer wanted a config file. Convert to JSON if you need it machine-readable, and leave the YAML alone if you do not.

The pair is worth it when the file is genuinely a list — exported records, fixtures, an event log somebody wrote by hand, a seed dataset — and the destination reads lines. In that case the conversion is instant, runs entirely in your browser, and the only thing left to check is the first line and the last.