Convert BZ2 to GZ

Bzip2 had a good decade. Between roughly 1998 and 2010 it was the obvious answer when gzip was not compressing enough, and a great deal of what was published in that window is still `.tar.bz2`: reference datasets, corpora, kernel and source tarballs, scientific data deposits, the fixture that a test suite has downloaded on every run since before anyone currently on the team joined.

The archive is not the problem. The problem is that something unpacks it on a schedule, and bzip2 decompression is slow enough to be visible in a build log. Nobody is going to re-cut the upstream release, so the practical move is to recompress locally into something that opens quickly and cache that instead. That is a different reader from the one converting for storage, and it points at a different answer: gzip, not XZ.

The two formats decompress in structurally different ways and the gap is not a matter of implementation quality. Gzip’s output is built by copying literal bytes and copying earlier runs from a 32 KB window — the inner loop is essentially memory movement, and modern hardware is extremely good at it.

Bzip2 has to invert a Burrows–Wheeler transform for every block of up to 900 KB. That means reconstructing the original ordering from a sorted permutation, which is a scattered, cache-unfriendly pass over the whole block before a single output byte of it is final, plus undoing the move-to-front and Huffman layers around it. It is real computation rather than data movement, and no amount of tuning turns it into memory speed. On an archive of any size the difference is minutes against seconds.

Expect the GZ to be larger, and expect the margin to depend entirely on the contents. Bzip2 sees up to 900 KB of context at a time where gzip sees 32 KB, so on prose, logs, CSV, source code and XML it finds repetition gzip cannot reach and the gap is real. The registry records exactly those two numbers, and they are the whole explanation.

On other contents there is no gap worth measuring. A bzip2 archive of JPEGs, PNGs, MP3s, MP4s or an already-compressed database file is barely compressed to begin with — the redundancy was removed by those formats before the archiving started — so recompressing it as gzip costs almost nothing and buys the full speed benefit. If that describes your archive, this conversion is close to free and there is nothing to weigh.

A gzip stream stores the original file name in its header. A bzip2 stream stores nothing of the sort — no name, no path, no hint about what it holds. That asymmetry is normally a curiosity and it produced a genuine defect here, which is recorded in the converter’s source.

The repack has to unwrap the TAR that sits inside a compressed archive, and the first version of the code decided whether it had found one by looking at the file name. From a GZ that worked, because gzip had remembered `output.tar`. From a BZ2 or an XZ the same tarball arrived called `input`, the check missed it, and the conversion produced an archive containing one opaque file instead of the thirty that went in — a success that returned the wrong thing. The fix was to force the tar type rather than infer it, which is why a bzip2 source is handled correctly today.

Ubiquity is the second half of this conversion’s argument and it is easy to underrate. DEFLATE is the compression inside HTTP `Content-Encoding: gzip`, inside PNG, inside a ZIP file and inside a Git object. Every mainstream language has it in its standard library — Python’s `gzip`, Go’s `compress/gzip`, Java’s `GZIPInputStream`, Node’s `zlib` — with no dependency to add and no licence question to answer.

Bzip2 is a separate library that has to be present, and in a minimal container, a locked-down runtime, a browser or a cloud function it often is not. Gzip is a registered HTTP content coding and bzip2 never was, so every HTTP client already decompresses one and none of them decompresses the other. If the archive is going to be read by code rather than by a person at a shell, gzip removes a dependency as well as a delay.

It is worth naming what you are giving up rather than only what you gain. Bzip2 compresses each block independently, so damage to one block does not automatically destroy everything after it, and recovery tools can walk a corrupted file and salvage the intact blocks. A gzip stream has no such structure — a flipped bit part way through generally ends the archive there.

How much that matters depends on where the file lives. On a modern filesystem with checksums, on object storage that verifies what it returns, or anywhere with a second copy, it matters very little. On a single ageing disk holding the only copy of something irreplaceable, it matters more — and in that case the right answer is neither format but a real backup with per-file integrity data. Recovery-friendliness is a poor substitute for redundancy.

If the archive is being kept rather than read, gzip is the wrong destination and XZ is the right one — it will compress the same contents considerably better than bzip2 did and still decompress faster. The reader on this page is not that person, but plenty of people converting a bzip2 archive are, and it is worth being told so.

The test is simple: count how often the archive is unpacked. Frequently, by machines, in a step somebody is waiting on — gzip. Rarely, by a person, from a bucket that charges by the gigabyte-month — XZ. The only case where bzip2 should stay is a machine with very little memory, because bzip2’s fixed 900 KB blocks need almost nothing to unpack where XZ needs a buffer scaled to its dictionary.

On our server rather than on your device. Most tools on this site convert locally and say so; archives are one of the exceptions, because the work needs 7-Zip and gzip as real programs. The file travels over an encrypted connection to a container that runs them and that has no outbound internet access of its own.

The job gets a scratch directory on a memory-backed filesystem, deleted the moment the job ends whether or not it succeeded, and it is killed after sixty seconds — which on this pair is a real constraint, since bzip2 is slow to decompress and a large archive of text is exactly the input that takes its time. The free tier accepts 25 MB per uploaded file, and before anything is extracted the archive is asked what it claims to expand to, with a declared total above 2 GB refused outright.

This conversion is worth doing only if the numbers say so, and getting them is a two-minute job: time the extraction of the bzip2 archive, time the extraction of the gzipped one, and put the two file sizes beside them. If the archive is small or the pipeline unpacks it once a month, the saving is noise and the change is not worth the churn of updating whatever points at the file.

Keep the bzip2 original until something has actually consumed the new file successfully. Not because the conversion is lossy — it is not, every member comes out byte for byte — but because the tarball is rebuilt rather than passed through, so member order, timestamps and permission bits can shift, and a build that depended on one of those fails in a way that looks nothing like an archive problem. Compare `tar tjf` on the source with `tar tzf` on the result first, and retire the original once they agree.