Convert Parquet to TSV

It is a binary container. `head` on one prints the four bytes PAR1 and then a screenful of noise, `grep` finds nothing reliable because the values are compressed and dictionary-encoded, and `wc -l` reports a number with no relationship to the row count. The file is designed to be read by an engine, and if the machine has no engine on it the file is opaque.

That is the situation this conversion is for: a Parquet extract on a server, a laptop without admin rights, a container with nothing installed, or a file somebody sent with a question that needs answering in the next two minutes. Turning it into lines is the fastest route to being able to answer anything at all about it.

Values in a data extract are full of commas — addresses, product names, free text, numbers written in a European locale. A comma-separated file has to wrap all of those in quotes, and every consumer of the file has to implement quoting, escaped quotes and embedded newlines correctly before it can find a column boundary.

Tabs are almost absent from that kind of data, so the delimiter and the content stop competing. After this conversion `cut -f3` finds the third column, `awk -F"\t"` finds it, and `sort -t$'\t' -k2` sorts on the second one. Nothing has to parse; things only have to split. That is the entire difference in reliability between the two delimiters, and it is why a tab-separated dump can be handled correctly by a one-line shell command.

A tab inside a value is the exception. It is rarer in data that came out of a Parquet file than in data that came out of a spreadsheet, but it happens — free-text fields carry whatever was pasted into them originally. When it does, the converter writes that field wrapped in double quotes, because the alternative is a row that silently grows a column.

That output is correct and a real parser handles it. `cut -f` does not: it counts tab characters and has never heard of a quotation mark, so from that row onward it sees one field too many and everything after it is off by one. If a pipeline is producing misaligned results on a handful of rows out of a million, `grep -c '"' output.tsv` will tell you in a second whether this is why.

A timestamp column is written as an ISO 8601 string — `2024-03-11T09:30:00.000Z` — which is the only sensible answer in a format with no date type. It has a property that matters a great deal in a shell: it sorts chronologically under a plain lexical sort, because the fields run from most significant to least.

`sort -k4` on a date column therefore does what you meant, `uniq -c` on the first ten characters counts rows per day, and a range filter is a string comparison. None of that works on a locale-formatted date, and none of it works on an epoch number without arithmetic. It is worth noticing that this is one of the few places where converting to text makes an operation easier rather than harder.

Parquet holds lists, structs and maps natively. A line of text does not, so a nested column is written as JSON inside its field: a list arrives as `["a","b"]`, a struct as an object with its member names.

That is deliberately the format that something else can read. `cut -f5 output.tsv | jq ".[0]"` pulls the first element of every list, and the nesting survives the trip rather than being flattened into columns whose names you would then have to work out. Since the field contains no tab, no quoting is added around it and the pipeline stays simple. Binary columns without a text annotation are written as lowercase hexadecimal for a related reason: it is reversible, it contains no delimiter, and nobody will mistake it for prose.

A null is written as an empty field. Every row keeps the same number of tabs, so the file stays rectangular and positional tools stay aligned, which is the behaviour you need.

It also erases a distinction the source file was careful about. Parquet records nullability per column and distinguishes a null from an empty string; in the output both are the same zero characters between two tabs. If you are counting missing values, `awk -F"\t" '$3==""'` counts both together and no shell command can separate them again. Where that matters, get the count from the source with a query engine rather than from the text.

Everything the Parquet file knew about its columns — the type of each, the row count, the per-column statistics — lives in a footer, and none of it survives into a tab-separated file. The output has a header row of names and after that it is characters.

If you have any way to read the schema, read it before converting and write it down. `DESCRIBE SELECT * FROM "file.parquet"` in DuckDB prints it in one statement. Where you have nothing at all, the text itself is the only evidence available, and the sensible move is to check a few distinct values per column with `cut -f2 | sort -u | head` rather than assuming that a column of digits is a quantity.

Parquet is compressed and dictionary-encoded, so the text it expands to is routinely several times the size of the file and frequently much more than that. A column of repeated status codes cost almost nothing in the source and costs a full copy per row in the output.

That is worth estimating before converting a file you have not looked at, because the expanded table is held in memory during the conversion. A hundred-megabyte extract can produce a great deal more than a hundred megabytes of text, and the ceiling here arrives sooner than the file size on disk suggests.

The Parquet reader is a library loaded on demand by this page and the writing is plain JavaScript. No request carries the file, so an extract from the middle of a data platform — usually the unaggregated, unmasked version — does not become a copy on somebody else’s server in order to be read.

There is no queue and no account, and the ceiling is 100 MB per file. For a file large enough to be a problem, the answer is a query engine rather than a bigger converter, and it is a better answer anyway because it lets you take only the columns you were interested in.

If you can install anything, install DuckDB. It reads the Parquet file in place with no import step, prints the schema in one statement, answers the question you actually had with a WHERE clause, and never materialises the whole table. Converting to text is strictly worse whenever that option exists.

This conversion is for when it does not: a locked-down machine, a colleague’s laptop, a file that arrived by email, a five-minute question that does not justify a setup. It is also the right choice when the destination genuinely is a text pipeline — an existing `awk` script, a diff against yesterday’s dump, a paste into something that takes tab-delimited input.