Introduction
Many of us who use Linux, whether as developers or casual users, have encountered something interesting when downloading files from the Internet: most of the files for Linux are in the tar.gz format. To me it was just a weird thing that was there, and a bit confusing. What was always bothersome was the lack of possibility to just double click it like a regular Windows executable. I was missing that run and go feeling. This article will talk a bit about this file format, what its historic roots are, and why and how it has held up to present time.
I will add that I generally learn about things only when really necessary, or when I am curious about them. After some time of just copying the commands I needed and googling them, I decided to just learn the tool I was using and understand it well enough to explain it to others. I find it fitting to mention this before we begin.
What Is tar.gz?
Simply put, tar.gz is just two stacked file formats, and it entails two things: firstly the file is an archive file, and secondly that this archive file has been compressed to be reduced in size. The tool used for archiving is called tar (Tape Archive) and the tool for compressing is called gzip (GNU Zip). Both of these operations can be done using only tar, but for the sake of explaining this in detail, I will explain both ways to go about this.
Archiving with tar
Archiving is the process of taking a directory containing multiple files and subdirectories, and organizing them into one file. Those files being written and read by tar are called tarballs.
It is also important to add that tar not only makes all of the files into one compact format, but it keeps the directory hierarchy of the files we are trying to archive. tar does this using a recursive tree walk, remembering the structure of directories and files inside.
Historically, before all the shiny electronic parts of the computer, people used physical tapes to insert into a computer and read out data. Conceptually tar does the same thing, but with everything mapped to software. It takes all of the contents of a directory, or just a single file, and organizes them as one stream - basically a file.
The not so great part about all of this is that these archive files are huge with regards to sending them across a network. They take the data as is, just make it into one stream and send it out where needed. Sending data in general is always a costly process, so it would be nice to find a way to reduce the size as much as possible, without harming the integrity of the data itself.
Compressing with gzip
This is where a compressor comes into play. A compressor takes a file and tries to reduce redundant appearances of repeated parts of that file. The contents of files are mapped and viewed as patterns, which in turn are marked if they are the same, in order to reduce the memory space required to store the same data many times. Basically it is a mechanism of pointers to places where frequently used "phrases" are used. In compression, offsets are usually used to the last seen place of a certain binary phrase.
I will add that this is the most general explanation, and one that relies on how text files are generally compressed, which is lossless compression. When we talk about image files or other formats, we get into the topic of tradeoffs that lossy compression brings to the table.
For the techniques by which compression is done using gzip, the algorithm used for lossless compression is called DEFLATE. It uses a combination of two algorithms:
- LZ77 identifies and replaces repeated occurrences of data with references to a single copy of that data existing earlier in the uncompressed data stream.
- Huffman Coding creates a table of all the possible symbols in a certain input stream with their estimated frequencies, and assigns weights that tell us how many times we are met with each symbol. The most common symbols are represented with the fewest number of bits, since we want the most frequent data to be as compact as possible.
Just to be clear, a symbol is an atomic token that can represent literal bytes, length codes, or distance codes.
Compressing data is really important, since it reduces file size without altering its original contents. Lossless formats can be rebuilt to their original format using gzip as well.
Why Are tar and gzip Used Together?
Now we get to the reason why tar and gzip are used together in the tar.gz format. When we have a directory that contains multiple files and subdirectories and we want to compress them, we cannot do that directly, as we cannot compress a directory. That is where a tape archive comes in, as it makes the directory's contents work as a single file. Using tar we now have a single file that we can then compress. This is the whole magic behind it.
Practical Examples
Now I will give a small example of how these tools work on a directory containing a file. We will look at two ways to do this: firstly doing both archiving and compression using tar alone, and then doing the same two operations with their own respective tools, tar and gzip.
First, open your Linux terminal and create a directory called testingtars, inside of which
we will create a text file called textfile.txt.
>mkdir testingtars
>cd testingtars
>touch textfile.txt
Method 1: Archive and Compress Using Only tar
>tar -czf textfile.tar.gz textfile.txt
The three flags -czf do the following: c creates a new archive, z compresses using
gzip, and f specifies the following arguments as the archive filename. The first argument is
the file we want to create, and the second is the file or directory we want to archive and
compress.
To decompress and extract using the same method:
>tar -xzf textfile.tar.gz
This decompresses, extracts, and unarchives in a single command.
Method 2: Archive with tar, Then Compress with gzip
For the sake of not overwriting existing files, we will create another text file called
text.txt.
>touch text.txt
>tar -cf text.tar text.txt
>gzip -k text.tar
The -k flag tells gzip to keep the original input file after compression, so the tar file
will be kept. The file we would get is text.tar.gz.
To decompress and extract, first use gunzip or gzip to decompress:
>gunzip text.tar.gz
or
>gzip -dk text.tar.gz
Then use tar to extract the archive:
>tar -xf text.tar
The -d flag stands for decompress, and -k keeps the tar.gz file.
Why Have tar and gzip Stood the Test of Time?
Looking at tar and gzip, one could ask how these two have stood the test of time and are still among the most used tools. The simplest answer would be: "It just works." The Unix philosophy is to do one thing well, and that is why reliability is such a big part of it. The file formats are stacked on top of each other, showing clear modularity. We literally see two distinct parts of one system working together.
It is also worth adding some interesting traits of tar that really show why it excels. One is
its flexibility with compression algorithms, instead of using the gzip flag -z, we can
simply opt for a different flag like -j for bzip2 or -J for xz. Another strength
is that tar stores metadata of the bundled files, including executable permissions, ownership,
and timestamps. Compare this to zip (common on Windows), which does not store metadata and
compresses on a per-file basis rather than on the archive as a whole.
Unill next time!!!
Whenever I see something I don't understand, the first principle is to understand it. Abstraction is great and helps out a lot, but what I personally like even more is unveiling the abstraction and seeing what is behind it all, what the smaller pieces do behind the scenes.
If you enjoyed this article and found it educational, I am very glad to hear that, and hope we will meet in another article soon. Cheers!