Convert XLSX to Parquet

Every other target a workbook has stores one record after another. Parquet stores one field after another: all fifty thousand order dates together, then all fifty thousand cities, then all the amounts. That is not a formatting preference, it is the whole reason the format exists.

The consequence shows up the first time you query it. A question that reads two fields out of two hundred touches two columns’ worth of bytes rather than every row, and a column of repeated values compresses far harder than the same values scattered through rows. Converting a spreadsheet into it is therefore a transposition as much as a serialisation, and the interesting work is deciding what type each column is. The file also carries its own schema in a footer, which is why a query engine can tell you the column names and types without reading a single row — something no delimited file and no spreadsheet can do.

The type is inferred from the values rather than declared, because a spreadsheet does not carry a schema. Each column is examined in full: if every non-empty value is a boolean it becomes BOOLEAN; if they are all whole numbers inside the 32-bit range it becomes INT32; if they are numbers but not all integers, or not all small enough, it becomes DOUBLE; anything else becomes a string.

Nulls do not participate. A column of numbers with gaps is still a numeric column, with the gaps written as nulls, which is what a query engine expects. The inference reads the whole column before deciding rather than sampling, which costs a pass over the data and removes an entire class of surprise. Sampling is what most tools do, and it is why a load that worked on a test file fails in production: the first thousand rows were clean and row forty thousand was not. Reading everything is slower and cannot be wrong in that particular way.

A column of postcodes that begins with five thousand numeric entries and then contains SW1A 1AA is a string column, and the numeric entries are written as strings too. The same is true of a quantity column with "n/a" in it, or an ID column where somebody typed a note.

The tempting alternative — take the type from the first row and null out whatever disagrees — produces a file that is valid, loads without complaint, and has quietly deleted the values that did not fit. Nobody discovers that until a count comes back short. A string column is visible, castable in one expression, and never silently loses a row. If a column arrives as text and you expected numbers, the fastest way to find out why is to query the values that fail to cast. That is usually a handful of rows with a footnote, a total, a stray unit symbol or the word "none" in them, and each of those is a genuine fact about the spreadsheet.

Parquet has a 64-bit integer type and it is not used here. The values arrive from the workbook as JavaScript numbers, which hold 53 bits of integer precision, so anything already in the pipeline has been through that limit before the writer sees it.

Writing such a value as INT64 would promise an exactness the number no longer has, and the failure would be invisible: an order reference off by one looks like an order reference. DOUBLE is the honest declaration of what is actually known. If your data has identifiers beyond nine quadrillion, store them as text in the spreadsheet before converting and they will arrive as a string column, intact.

On a sheet of fifty thousand rows and six columns — an order id, a product code, a city, a quantity, a price and a flag — the Parquet file came to 301 KB. The same data was about 4.3 MB as an .xlsx and 1.8 MB written out as tab-separated text.

The gap is not compression alone. Repeated values in a column store are held once and referenced, which is why the city and product code columns cost almost nothing, while a column of unique free text would narrow the difference considerably. Expect a large saving on operational data and a modest one on a sheet that is mostly distinct prose. The saving compounds when the file is queried rather than merely stored, because a query that reads two of six columns pays for two of them. Against a spreadsheet, where the only way to answer any question is to load the entire workbook, that is the difference the format was designed to make.

What a spreadsheet stores in a date cell is a number and a display format. The conversion writes the number, so 1 January 2024 becomes 45292, counted from the end of December 1899, and the column is typed as an integer like any other whole number.

That means there is no timestamp semantics in the output, and a query engine will treat the column as what it is: an integer. Applying the calendar is a one-line expression where you query — adding the day count to the epoch date — and doing it there is better than reformatting the workbook to text first, which replaces an arithmetic problem with a parsing one.

Nothing special is required. DuckDB reads the file directly in a FROM clause, pandas reads it with a single call, and Spark and the query engines built on it treat it as a native table format. The file begins and ends with the four bytes PAR1, which is how any of them recognise it before reading the footer.

The first thing worth doing after loading is looking at the inferred types rather than the first ten rows. A column you expected to be numeric and that arrived as text is telling you something true about the spreadsheet, and it is far cheaper to learn it now than after the file has been joined to three others. Casting on the way into a table is one expression per column and is the normal thing to do. What you should not do is cast blindly: a column that came through as text for a reason will silently produce nulls where the values did not fit, which is exactly the failure the writer refused to commit on your behalf.

The conversion reads the first sheet of the workbook. Parquet holds one table with one schema, so a multi-sheet workbook cannot be represented in a single file without inventing a merge that nobody asked for, and taking the first sheet is the only choice that does not.

For a workbook where several sheets matter, convert each one separately and let the query engine put them back together. That is the arrangement those engines are built for — several files, read together, filtered on load — and it produces a better result than a single sheet with a discriminator column bolted on. It also keeps the schemas honest. Three sheets that look alike in a workbook frequently differ by a column or a type, and forcing them into one file means deciding which of those differences to erase — a decision better made explicitly in a query than implicitly by a converter.

Both halves run in your browser: the spreadsheet reader and the Parquet writer are libraries loaded on demand by this page, and no request carries the workbook anywhere. For an export of customer or financial data that is frequently the deciding factor, because uploading it to a converter would be the part your policy actually forbids.

The limit is memory rather than a plan tier. The sheet is read into rows, transposed into columns and written, so the whole table exists at once; tens of megabytes is routine and hundreds is where a browser tab begins to strain. Past that point a script with a streaming reader is the right instrument, and saying so beats failing in the middle of a large file.

Parquet is a poor choice for anything a person has to look at. It is binary, it is not editable, and a colleague without the right tooling cannot open it at all — the registry marks its support as patchy for exactly that reason. If the file is going to a human being, send a spreadsheet or a CSV.

It is also the wrong target for a one-off load, where the writing cost buys nothing: NDJSON streams into an ingest job and is done. Choose Parquet when the same data will be queried repeatedly, joined against other tables, and kept — which is precisely when the size and the typed columns start paying for themselves.