What is a Parquet file?

Parquet is a binary format, meaningful only to a program that knows it. It is used for archiving and moving data between programs.

The extension is .parquet, and the full name is Apache Parquet. Both matter less than what the file can hold, which is what the rest of this page is about.

Apache Software Foundation published it in 2013.

Age is worth knowing here for one practical reason: the older a format is, the more programs have had time to learn it.

It is published in full, so anyone can implement it from the document rather than by inspection, which is why it turns up in so many programs and why files written twenty years ago still open. A published specification is not the same thing as a royalty-free one: where a format wraps a codec, the patent licensing is a separate question the standard does not settle.

Parquet stores its content exactly. Saving it again changes nothing, so it can be opened, edited and re-saved as often as you like without accumulating damage — which is what makes it a working format rather than a delivery one.

Parquet carries a checksum, so a damaged file is detected rather than silently mis-read and optional encryption.

An encrypted file has to be unlocked before anything can convert it, here or anywhere else — a password is not something a converter can work around, and one that claimed to could not be trusted with the file either.

pandas, Apache Spark and DuckDB read it, and so do most programs of the same kind.

If a file will not open, the format is rarely the problem — it is more often that the program predates it. Converting to something older is the reliable way past that, and it is what the rest of this site is for.

No browser reads it.

That is the single most common reason to convert it: not that the format is bad, but that the place you want to show the file cannot read it.

Parquet is meant to be handed over rather than worked in. Editing one is possible and rarely pleasant; the sane approach is to change the source and export again.

The recurring complaints: support outside its home ground is uneven.

None of these is a reason to avoid the format. They are the things worth knowing before you are surprised by one, which is a different claim and a more useful one.

A CSV writes one record at a time: name, date, country, amount, then the next record. Parquet writes one column at a time: every name together, then every date, then every country.

That single rearrangement is the whole format. It sounds like a filing preference and it changes the economics of everything downstream, because a query that wants two columns out of fifty can read exactly those two and skip the rest, and because a column of similar values compresses far better than a row of unrelated ones.

Values in a column are alike — dates look like dates, country codes repeat, amounts share a magnitude — and that similarity is what compression feeds on. Parquet also encodes cleverly before compressing: a column with few distinct values becomes a dictionary and a list of small integers, and a run of identical values becomes a count.

The result is routinely five to ten times smaller than the same data as CSV, and occasionally far more for wide tables with repetitive columns. That is a straightforward saving on storage and, more importantly, on the transfer time of everything that reads it.

A CSV has no types at all, so every consumer guesses — and that is where leading zeros vanish and identifiers turn into dates. Parquet records the type of every column in its own schema: this is a 64-bit integer, this is a string, this is a timestamp with a time zone, this is a decimal with that precision.

So the data arrives meaning what it meant. Nothing is inferred, nothing depends on a locale, and a column of postcodes stays a column of postcodes. For anything moving between systems, this is a larger practical benefit than the compression.

Parquet is organised into row groups, and each one records statistics per column — the minimum, the maximum, how many nulls. A query filtering on a date range can read those statistics, see that a whole row group falls outside the range, and skip it without decompressing anything.

That is why a query over a large Parquet dataset can be dramatically faster than the same query over CSV, beyond the reading of fewer columns. It also explains why partitioning a dataset by date into a directory structure — a folder per day — works so well: the engine skips entire files before it starts.

Reading one record. Reconstructing a single row means gathering a value from every column, which is precisely the access pattern the layout is not built for. Parquet is an analytics format, not a database.

Appending. A file is written whole, with its statistics and footer computed at the end, so adding a row means rewriting or writing a new file. Streaming data therefore lands as many small files and is compacted later, which is a normal part of running a data lake and a nuisance if nobody planned for it.

And being looked at. It is binary — a text editor shows nothing useful, and inspecting a file needs a tool.

The barrier is lower than it used to be. DuckDB reads a Parquet file directly with a single SQL query and installs in seconds. Python with pandas or Polars reads one in a line. Several free desktop viewers open one like a spreadsheet.

Converting to CSV is the other route, and the right one when a colleague needs the data in a spreadsheet — accepting that the types go back to being guesses and the file becomes several times larger. Keep the Parquet as the source and generate the CSV for the person who asked.

Anywhere data is analysed at scale: data lakes on cloud storage, Spark and Databricks, Snowflake and BigQuery exports, AWS Athena, dbt pipelines, and increasingly ordinary Python analysis where it has quietly replaced CSV as the default way to save a dataframe.

It is also becoming the format public bodies and research projects publish large datasets in, precisely because the alternative — a two-gigabyte CSV nobody can open in a spreadsheet anyway — serves nobody. If you have been handed one, that is almost certainly why.