Convert GZ to XZ

Nobody converts a gzipped file they are about to open. The reader here is looking at something that has accumulated: a year of rotated application logs, nightly database dumps kept for compliance, a research dataset that has to stay available and is read once a quarter, CI artefacts nobody has deleted. Gzip was the right choice when the files were written — it is fast, it is everywhere, and it was already in the log rotation config — and it is not the right choice for the second half of the file’s life.

That distinction is the whole decision. Gzip optimises for the moment of writing and reading; XZ optimises for the years in between. If the archive is going to be fetched and unpacked by something in a hurry, or by a device with very little memory, leave it as it is. If it is going to sit in cold storage and be read almost never, the conversion is close to free money, and it is worth doing across the whole folder rather than one file at a time.

The number in the registry is the entire explanation. Gzip, standardised as RFC 1952 in May 1996, uses DEFLATE with a 32 KB history window: when it is compressing byte one million, it can see back to byte nine hundred and sixty-eight thousand and no further. Anything that repeated earlier than that is compressed again from scratch, as though it had never been seen. XZ, from 2009, carries a dictionary the registry records at up to 64 MB — two thousand times the reach.

That is why the gain is enormous on some inputs and zero on others. A year of log lines repeats the same forty message templates for gigabytes, and gzip rediscovers each of them thousands of times while XZ learns them once. A SQL dump repeats column names on every row. Source trees repeat licence headers, imports and boilerplate across hundreds of files. Against that, a GZ holding a photograph or a video has no long-range repetition to find, because the redundancy was removed by JPEG or H.264 before gzip ever saw it, and the strongest compressor in the world cannot invent structure that is not there.

Compression time is the obvious cost and the least important one, because it is paid once by a machine that is not waiting for you. XZ is dramatically slower than gzip at the same task — that is the direct consequence of searching a dictionary two thousand times larger — and for a job you run overnight across a storage bucket, that is simply not a problem worth optimising.

Memory at decompression is the cost that actually decides things, and it is the one people miss. Gzip needs its 32 KB window to unpack, which is to say nothing at all; XZ needs a buffer scaled to the dictionary the file was written with. On a laptop or a server that is unremarkable. On a router, a small embedded device, a memory-capped container or a build agent with tight limits, it is the difference between the archive opening and the process being killed. Check the machine that will unpack the file, not the one that compresses it.

Both formats compress exactly one stream, so neither knows what a directory is. In practice a `.gz` is almost always a `.tar.gz`: a TAR carrying the names and the folders, with gzip wrapped around it. The conversion takes both layers off, then puts both back — the members are collected into a fresh TAR and that TAR is compressed with xz.

The case worth knowing about is the one where the source was not a tarball. A plain `access.log.gz` holds one ordinary file, and the repack still builds a TAR around it, so what comes back is a tarball containing `access.log` rather than a bare compressed file. That is harmless if you unpack it with `tar xJf` and confusing if you reach for `unxz` and find a file called `output.tar`. If your tooling expects a single compressed stream with nothing wrapped around it, this is the detail to plan for.

A gzip stream has a header field for the original file name, which is why decompressing `dump.gz` on a Unix machine can hand you back `dump.sql` even though nothing in the extension said so. XZ has no such field. It stores an integrity check and nothing about where the data came from.

That has a practical consequence at both ends of this conversion and it caused a real bug in this converter, recorded in the code: because gzip stores a name and xz does not, an inner tarball arrives from a GZ called something sensible and from an XZ called whatever the extractor chose. The fix was to force the tar type rather than infer it from the name. For you, the lesson is simpler — the extension you put on the downloaded XZ is the only label it will ever carry, so name it `something.tar.xz` and mean it.

Both formats verify themselves, which is one of the reasons this conversion is safe to run unattended. A gzip stream ends with a CRC-32 and the uncompressed length; an XZ stream carries its own check, CRC-64 by default, over the uncompressed data. In both cases a truncated download or a flipped bit produces an error on decompression rather than a file that quietly reads short.

What neither can do is tell you which file is damaged, because at that layer there are no files. A ZIP or a 7Z records a checksum per entry and can report that one member of forty is corrupt; a compressed stream can only report that the stream is corrupt. If per-member integrity is what you need from an archive — because you intend to recover the rest when one part rots — then a stream compressor is the wrong shape for the job regardless of how well it compresses.

This conversion does not run on your device. Most tools on this site keep the file local and say so; repacking archives is one of the exceptions, because it needs 7-Zip and xz as real programs. The file goes over an encrypted connection to a container that runs them, and the container has no outbound internet access of its own.

Each job gets a scratch directory on a memory-backed filesystem, and that directory is deleted as soon as the job ends regardless of whether it succeeded. Anything still running after sixty seconds is killed — a malformed archive puts a compressor in a loop far more often than it makes one fail. The free tier accepts 25 MB per uploaded file, which for a gzipped log archive is a great deal of text but is a real ceiling on a dataset.

BZ2 is the other traditional answer to "gzip is not compressing this enough", and on most text it now loses to XZ on both axes at once — a worse ratio and slower decompression. The registry marks BZ2 as legacy for that reason. Its one remaining advantage is the flip side of the memory point above: bzip2 works in blocks of up to 900 KB, so unpacking one never needs much memory at all.

So the choice is not really about ratio. Pick XZ when the machine that will unpack the archive is an ordinary computer, which it usually is, and pick BZ2 only when it is not — an old appliance, a constrained container, a system where xz is genuinely not installed and cannot be. Choosing BZ2 for its compression alone is a decision from about 2005 that has not been true for a long time.

Do one file first and look at the two sizes. A single representative archive tells you what the rest of the folder will do far more reliably than any rule of thumb, because the answer is decided by the contents rather than by the formats — the same conversion that halves a log archive does nothing at all to a folder of photographs. If the first file barely moves, stop; you have learned that the data has no long-range redundancy and no compressor is going to find any.

Then check the listing rather than only the size. Run `tar tJf` on the result and compare the entry count and the paths against what the GZ held, particularly the top level: an archive packed from inside a directory and one packed from its parent differ by a leading folder in every path, and that difference breaks scripts long after anyone remembers converting anything. Once the listing matches and the size is what you expected, the original has no job left to do.