As we all know. in a typical Hadoop environment we have to store and deal with large volumes of data which makes data compression a necessity. We all know that the slave nodes stores data in them, commonly each slave node or datanode has about 45TB of raw storage space available for HDFS.
Although Hadoop machines are designed to be inexpensive adding memory to the machines can significantly cost you. We can simply solve this problem by compressing a;; of our data.
Benefits of Data Compressing
- It saves a great deal of storage space.
- It also speeds up the transfer of the blocks throughout the clusters.
Obviously we have a variety of codecs to choose from depending upon our usage and scenario. If you don’t understand what codec is then, here’s a quick definition.
Codec = Compressor + Decompressor
It can be set a set of hardware or software that compresses and decompresses the data using a specific algorithm. We already know that the if we are ingesting a file to HDFS and it has size more than the default file block size (128 MB), it will automatically split the file into separate chunks. Here an important concept of Splittable Compression comes into play. Most codecs cannot decompress the file block splits of the files independently, because of which we have to allocate the job to a single mapper.
Now we cannot parallely process data that results in decrease of our performance; speaking of performance we have a trade-off between Compression degree, a compression, and space.
Generally if we have more degree compression then we would have to put more computation power or computation cycles in compression/decompression of the data. But more degree of compression saves us space.
Some of the common codecs supported by the Hadoop framework are as follows:
-
Gzip - A compression utility that was adopted by the GNU project. It’s file have an extension of .gz. You can use gunzip command to decompress the files.
-
Bzip2 - from the usability standpoint Bzip2 and Gzip are similar. But Bzip2 has much more degree of compression then the Gzip but it is also slower . You can use Bzip2 codec space priority is higher and the data will be rarely needed to be queried.
