Convert XZ to GZ

The reader here rarely made the file. An upstream project publishes a `.tar.xz`, and it now has to be served to somebody else: mirrored on an internal artefact store, attached to an internal release, dropped into a shared drive, put behind a download link for an audience whose machines and habits you do not control. Upstream chose XZ because they were optimising bandwidth for a technical audience. You are optimising for whoever clicks the link.

The other version is a runtime rather than a person. A data-loading step written in Python or Go opens gzip with a standard-library import and needs a dependency, a system package or a subprocess for anything else; a serverless function or a hardened container image often cannot have one. A build image somebody froze in 2014 has gzip and bzip2. In all of these the constraint is external, and the honest recommendation for anyone without such a constraint is to leave the XZ exactly as it is.

Gzip’s advantage is not technical, it is historical, and it compounds. DEFLATE was published as RFC 1951 in May 1996 and gzip’s container as RFC 1952 in the same month; the same algorithm sits inside PNG, inside every ZIP file, inside HTTP’s `Content-Encoding: gzip`, inside a Git object and inside a Java JAR. Any platform that does any of those things already has a DEFLATE implementation linked in, so exposing a gzip reader costs it nothing.

XZ, from 2009, had no such carrier. It is an excellent format and it has to be installed as itself. On Linux that is trivial, because the package manager depends on it; everywhere else it is a decision somebody has to make. The registry records browser support as `all` for GZ and `none` for XZ, which is a narrow statement about one context and a fair signal about many more. When you are handing a file to an unknown environment, that difference is the whole question.

The gap between the two formats is a single number and the registry carries both halves of it: gzip’s DEFLATE searches a 32 KB window, and XZ’s LZMA2 works with a dictionary of up to 64 MB. When gzip is compressing byte one million it can see back to byte nine hundred and sixty-eight thousand and no further; XZ can still see the beginning of the archive.

On archives of many similar files that is not a marginal difference. Two hundred source files sharing a licence header, a year of log lines drawing on the same forty templates, a documentation build repeating navigation on every page — XZ learns each pattern once and gzip relearns it constantly. Expect that to show up plainly in the two file sizes. Where it does not show up is on archives of JPEGs, MP4s or anything else already compressed, and if that is your archive then this conversion costs you almost nothing.

Gzip decodes from front to back with a fixed 32 KB window and no index, which is the source of its most useful property: a gzipped tarball can be consumed as it arrives. `curl https://… | tar xzf -` unpacks a download that never lands on disk, a web server can compress a response it has not finished generating, and a log shipper can compress on the fly without knowing how much data is coming.

XZ can be streamed too, but the memory it needs to do so is proportional to its dictionary rather than fixed, which changes the character of the thing. A pipeline where the consumer is small — an embedded device, a memory-capped container, a function with a hard limit — can consume a gzip stream of unlimited length and may not be able to consume an xz stream at all. If your reason for converting was that something failed to allocate memory, this is the paragraph that explains it.

The conversion extracts the archive and writes a fresh TAR from the contents before compressing it. Every file inside comes out byte for byte — this is a lossless repack — but the tarball is new, so member order, recorded timestamps and permission bits may differ from upstream’s and no published checksum will verify against the result.

For a mirror that is a matter of presentation rather than correctness. Publish your own hash of the file you are actually serving, and say plainly that the canonical artefact is upstream’s `.tar.xz` with upstream’s signature. What you must not do is serve a repackaged file under upstream’s hash or imply that the two are the same object. If the audience needs to verify against the project’s own signature, they need the original, and the right answer is to offer both rather than to replace one with the other.

One small thing improves in this direction. A gzip stream has a header field for the original file name and an xz stream has none at all — xz stores an integrity check and nothing about where the data came from. That asymmetry is why `gunzip dataset.gz` can hand you back `dataset.tar` while `unxz dataset.xz` leaves you holding a file called `dataset` that turns out to be an archive.

It is a courtesy rather than a guarantee: plenty of tools write a generic name into the field and plenty of others ignore it on extraction. The extension is still what will actually be read, so name the download `.tar.gz` before you publish it. Leaving it as `.gz` invites somebody to run `gunzip` and be left with a stray tarball, which is the exact confusion that sends people looking for a converter in the first place.

If the constraint is purely memory rather than availability, bzip2 is worth considering instead: it works in independent blocks of at most 900 KB, so it needs almost nothing to unpack, and it compresses better than gzip on text. The cost is decompression time, because undoing a Burrows–Wheeler transform is real computation rather than data movement.

The test is which failure you are actually working around. Something reported that it could not allocate memory, on a slow device with plenty of time — bzip2 is a reasonable answer. Something did not recognise the format, or you are handing the file to people rather than machines — gzip, without hesitation, because the registry marks BZ2 legacy and it is nowhere near as widely present as gzip. Choosing between the two on ratio alone is the wrong axis for both.

On our server rather than in your browser. Most of the tools on this site convert on your own device and say so plainly; archive work is one of the exceptions, because it needs 7-Zip, xz and gzip as real programs. The file travels over an encrypted connection to a container that runs them and has no outbound internet access of its own.

Every job gets a scratch directory on a memory-backed filesystem, thrown away as soon as the job ends whatever the outcome, and killed after sixty seconds because a malformed archive puts a compressor in a loop more often than it makes one fail. The free tier accepts 25 MB per uploaded file, and the archive is refused before extraction if it declares an expansion above 2 GB — a check that catches a good proportion of deliberately hostile files and, occasionally, an honest scientific dataset.

Where you control the download page, the best outcome is usually not a conversion at all but an addition. Offer the upstream `.tar.xz` for the audience that can use it and the `.tar.gz` beside it for the audience that cannot, label which is which, and let people choose. Bandwidth for the technical audience stays cheap and the awkward support conversation disappears, at the cost of a little storage.

Where you do have to pick one, pick by who is on the other end rather than by which format is better. A public download aimed at a general audience, an internal artefact fetched by tooling you have not audited, anything attached to a ticket for a customer — gzip. A release aimed squarely at developers on Linux, or an archive you are keeping rather than distributing — leave it as XZ, and this page has done its job by talking you out of the conversion.