Hadoop supports seven compression codecs you are likely to meet: DEFLATE, gzip, bzip2, LZ4, Snappy, Zstandard and LZO. They differ in how small they make the data, how much CPU they cost, and whether a compressed file can be split so that several tasks process it in parallel. For plain text files on HDFS, that last property matters most.
The short version: do not store large raw text files compressed with gzip or Snappy. Put the data in a container format such as Parquet, ORC or Avro, which compresses in blocks and stays splittable with any codec, and use Snappy or Zstandard inside it.
The codecs compared
| Codec | Hadoop class (org.apache.hadoop.io.compress.) | Extension | Splittable as a raw file | Ratio | Speed |
|---|---|---|---|---|---|
| DEFLATE | DefaultCodec | .deflate | No | Medium | Medium |
| gzip | GzipCodec | .gz | No | Medium | Medium |
| bzip2 | BZip2Codec | .bz2 | Yes | High | Slow |
| LZ4 | Lz4Codec | .lz4 | No | Lower | Very fast |
| Snappy | SnappyCodec | .snappy | No | Lower | Very fast |
| Zstandard | ZStandardCodec | .zst | No | High, tunable by level | Fast |
| LZO | com.hadoop.compression.lzo.LzopCodec | .lzo | Only after indexing | Lower | Very fast |
Ratio and speed are relative; the actual numbers depend on your data, so test with a sample of it.
Notes on individual codecs:
- gzip and DEFLATE use the same algorithm. gzip adds a header and checksum and is the format every other tool understands.
- bzip2 is the only codec here that is splittable as a plain file, and the slowest. It suits data that is written once and rarely read.
- Snappy and LZ4 trade compression ratio for speed and are the usual choice for intermediate data and for data that is queried often.
- Zstandard (zstd) gives ratios close to gzip or better at much higher speed, with a level setting to move along that trade-off. It has been available since Hadoop 2.9 and is now a common default for stored data.
- LZO is not shipped with Hadoop because its library is GPL-licensed. It comes from the separate
hadoop-lzoproject, which also provides an indexer that makes.lzofiles splittable. New systems rarely choose it.
Why splittability matters
HDFS stores a file as a sequence of blocks, 128 MB each by default. MapReduce, Hive and Spark normally create one input split per block and process the splits in parallel.
That only works if a task can start reading in the middle of the file. A gzip stream can be decoded only from its beginning, so a 10 GB .gz file becomes a single split handled by a single task, however many nodes you have. The job log shows it:
INFO mapreduce.JobSubmitter: number of splits:1
The task also reads most of its blocks over the network, since they live on different nodes. bzip2 avoids the problem because its format has block markers that a reader can find from any offset.
Compression format and file format are separate decisions, and combining them is the way out:
| What you store | Splittable? |
|---|---|
| Text or CSV, uncompressed | Yes |
| Text or CSV compressed with gzip, Snappy, LZ4 or zstd | No |
| Text or CSV compressed with bzip2 | Yes |
| SequenceFile or Avro with any codec | Yes |
| Parquet or ORC with any codec | Yes |
Container formats compress each internal block separately and keep an index of where blocks start, so a reader can begin at any block boundary.
Check which codecs your cluster can use
Some codecs depend on native libraries on every node:
hadoop checknative -a
Native library checking:
hadoop: true /opt/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
zstd: true /usr/lib/x86_64-linux-gnu/libzstd.so.1
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
The exact lines and paths vary by version and distribution. What to look for:
ZStandardCodecneeds the native Hadoop library built with zstd support. Ifzstdshowsfalse, jobs that use it fail.- From Hadoop 3.3.1, Snappy and LZ4 use bundled Java libraries (
snappy-javaandlz4-java) and need no native installation. Older versions require native libraries for both. - gzip and bzip2 fall back to slower pure-Java implementations when native libraries are missing.
Enable compression in a MapReduce job
There are two places to compress, with different goals.
Intermediate (map) output is shuffled across the network between map and reduce tasks and then discarded. Compressing it with a fast codec reduces shuffle I/O and is almost always worthwhile. Splittability is irrelevant here.
Final output is what lands in HDFS. Choose the codec according to how the files will be read.
Both can be set per job with -D properties. This runs the bundled word count example over /data/logs:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount \
-D mapreduce.map.output.compress=true \
-D mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec \
-D mapreduce.output.fileoutputformat.compress=true \
-D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec \
/data/logs /data/wordcount-out
Verify the result:
hdfs dfs -ls /data/wordcount-out
hdfs dfs -text /data/wordcount-out/part-r-00000.gz | head
The output files carry the codec's extension (part-r-00000.gz). hdfs dfs -text picks the codec from that extension and prints decompressed content, whereas hdfs dfs -cat prints the raw compressed bytes.
To make these the defaults for every job, set the same properties in mapred-site.xml. To undo, set the two compress properties to false; existing files stay as they are and remain readable.
When the output is a SequenceFile, also set mapreduce.output.fileoutputformat.compress.type=BLOCK. The default, RECORD, compresses each record separately and achieves much less.
On input you configure nothing. Hadoop picks the decompressor from the file extension, using the codecs listed in io.compression.codecs.
Compression in Parquet and ORC
Most data that used to sit on HDFS as delimited text is now stored as Parquet or ORC, on HDFS or on object storage. Both are columnar: values from one column are stored together, encoded (dictionary and run-length encoding, for example) and then compressed by a general-purpose codec. Similar values next to each other compress far better than mixed rows.
Parquet supports uncompressed, Snappy, gzip, LZO, Brotli, Zstandard and LZ4_RAW. The older LZ4 codec name is deprecated because of an incompatible framing format. ORC supports zlib, Snappy, LZO, LZ4 and Zstandard.
Defaults differ between engines and versions. Spark writes Parquet with Snappy unless told otherwise, and the ORC library's own default changed from zlib to Zstandard in ORC 2.0, while engines that bundle ORC may apply a different default of their own. Set the codec explicitly if it matters to you.
In Spark SQL:
CREATE TABLE events_zstd
USING parquet
OPTIONS (compression 'zstd')
AS SELECT * FROM events;
In Hive, for an ORC table:
CREATE TABLE events_orc
STORED AS ORC
TBLPROPERTIES ('orc.compress' = 'SNAPPY')
AS SELECT * FROM events;
The codec is recorded in each file's metadata, so readers need no configuration, and files written with different codecs can sit in the same table.
Which codec to choose
| Situation | Reasonable choice |
|---|---|
| Map output and other temporary data | Snappy or LZ4 |
| Parquet or ORC tables queried often | Snappy or Zstandard at a low level |
| Parquet or ORC archives read rarely | Zstandard at a higher level, or gzip |
| Raw text that must stay as text and be processed in parallel | bzip2, or split into many gzip files of roughly one block each |
| Files exchanged with systems outside Hadoop | gzip |
Common mistakes
- One huge gzip file. It is processed by a single task. Convert it to a container format, or write many smaller files.
- Expecting
.snappyor.lz4files to be portable. Hadoop wraps these codecs in its own block framing. Files written bySnappyCodecorLz4Codecare generally not readable by the standalone command-line tools, and the reverse is also true. - A codec missing on some nodes. A job that works on one node and fails on another with a native-library or "codec not found" error usually means an inconsistent installation. Run
hadoop checknative -aon each node. - Compressing data twice. Running gzip over Parquet, ORC, JPEG or already-compressed archives burns CPU for almost no gain.
- Many tiny compressed files. Each file costs NameNode memory and a task of its own. Compression does not fix a small-files problem.
- Choosing by ratio alone. A tighter codec that makes every query slower is often a bad trade. Storage is usually cheaper than compute.
Further reading
Get the weekly commit
New database deep dives every week.
