Convert XML to NDJSON

Newline-delimited JSON needs something to split on, and the split here happens at a list on the outside of the parsed data. Parsing XML never produces one: the result is always an object with a single key, the root element name, holding everything else. One value in, one line out.

That holds however large the file is and however many repeated elements it contains. A 40 MB export of ten thousand records converts into a single 40 MB line. Nothing is lost and nothing is split, and if what you needed was ten thousand lines, this conversion has not done it.

The route that works is two steps. Convert the file to JSON, then run it through jq: select the path holding the repeated element, iterate it, and use the compact output flag so each object lands on its own line. It is one command, it is repeatable in a script, and it puts the choice of what counts as a record where it belongs — with you.

No converter can make that choice from the document alone. In an RSS feed the record is the item element; in a SOAP response it might be a row three levels inside the body; in a bank export it is whatever the vendor decided to call a transaction. A tool that guessed would be right often enough to be trusted and wrong often enough to corrupt a load.

This is not a pair with no use. If your inputs are many small XML documents rather than one large one — a directory of invoices, a folder of manifests, a batch of per-event payloads — then one line per file is precisely the shape a loader wants, and concatenating the outputs produces a valid NDJSON stream with one record per source document.

It also works for inventory. Converting each config file in a repository and appending the lines gives you a queryable dataset of what those files contain, which is a genuinely useful thing to have and is hard to get any other way. In both cases the conversion is doing its job and the single line is the feature.

Even after you have split the records, one XML behaviour follows them. A repeated element becomes an array only when it repeats: a record containing one tag element produces a string, and a record containing two produces a list of strings. Within a single export, some lines will have the array and some will not.

A schema-on-read store will infer a type from the first records it sees and then reject or coerce the rest. The fix belongs in the jq pass rather than in the loader: normalise every field that can repeat into an array as you emit each line, so the stream is uniform before anything downstream sees it.

Compact JSON with no indentation, keys in document order, terminated by a newline. Attributes appear as keys prefixed with @ and element text on a tag that also has attributes appears under #text. A namespace prefix stays inside the key name, so soap:Body is a key with a colon in it, and a document with an XML declaration carries an extra top-level key named ?xml.

Those last two are worth stripping before a load. Field names containing colons and question marks are awkward or invalid in a good number of query engines and table schemas, and the declaration is metadata about the file rather than data from it.

Values that look numeric are parsed, in attributes and element text alike. That is convenient for a count and destructive for identifiers: an order reference written 007 arrives as the number 7, and a version attribute written 1.0 arrives as 1. In a data store those become numeric columns and the padding is gone from every row.

One case behaves better than you might fear: an integer too long to survive as a JSON number is left as a string rather than rounded, so a nineteen-digit reference comes through intact. Scientific notation does not get that protection — a value written 1e3 arrives as 1000. Nothing raises an error either way. If the export contains reference numbers you will join on later, cast them back to strings in the jq pass, before the first load rather than after.

NDJSON is read directly by jq, by pandas with its lines option, and as a load format by the common warehouse loaders. Elasticsearch bulk needs an action line before each document, which is added in the same jq pass that splits the records.

The thing to plan for with this pair specifically is line length. A reader that processes a line at a time has to hold that entire line in memory, so a whole-document line is a whole-document buffer, and any per-record size limit in the destination applies to the complete file rather than to a record. That is the practical reason to split the records before loading rather than after.

If the export is large, arrives regularly, and has to become a stream every time, the honest recommendation is not a browser converter at all. A streaming parser reads an XML file element by element without building the whole tree in memory, and emitting one JSON line per record from that loop is a short program in any language.

That approach also survives files bigger than memory, which nothing on this page does — parsing here builds the entire document as JavaScript objects before a single byte is written. For a one-off file, or for the per-file cases above, the converter is faster than writing the program. For a recurring pipeline it is not.

XML comments are dropped during parsing and NDJSON has no comment syntax to receive them, so any documentation inside the export is gone. There is also no header, no schema and no preamble: every line in an NDJSON file is a record, and a loader will try to read anything at the top of the file as one.

If the load needs to record where the data came from, that belongs in a field on the record or in the filename. Adding it as a first line makes the file invalid for its purpose, and the loader will tell you so in a way that is much less clear than this paragraph.