How to Ingest Data in Hadoop

• Files are write-once. You can append to a file and delete it, but you cannot modify bytes in the middle. Updates are handled by rewriting files, or by a table format such as Apache Iceberg or Hive ACID tables layered over them.
• -put does not transform anything. The bytes arrive as they were. Converting to a columnar format, cleaning and partitioning happen in a later job.

While an upload is in progress, the shell writes to a temporary name ending in ._COPYING_ and renames it at the end, so readers of the directory never see a half-written file under its final name. Follow the same pattern in your own tools: write to a staging directory and move the finished file into place with hdfs dfs -mv, which is an atomic rename within HDFS.

Verify and undo

hdfs dfs -ls /data/raw/orders
hdfs dfs -du -s -h /data/raw/orders
hdfs dfs -count /data/raw/orders
hdfs dfs -cat /data/raw/orders/orders_2026-10-01.csv | head -5

-count prints the number of directories, the number of files and the total bytes. Compare the byte count with the source.

To remove an upload:

hdfs dfs -rm -r /data/raw/orders/orders_2026-10-01.csv

If trash is enabled on the cluster (fs.trash.interval greater than zero), the file moves to .Trash in your HDFS home directory and can be moved back until the interval expires. Adding -skipTrash deletes it immediately and permanently.

Upload over HTTP with WebHDFS

When the sending machine has no Hadoop client installed, use the WebHDFS REST API on the NameNode's HTTP port (9870 by default in Hadoop 3). Creating a file takes two requests. The first asks the NameNode where to write:

curl -i -X PUT "http://namenode.example.com:9870/webhdfs/v1/data/raw/orders/orders_2026-10-02.csv?op=CREATE&user.name=etl"

HTTP/1.1 307 Temporary Redirect
Location: http://datanode3.example.com:9864/webhdfs/v1/data/raw/orders/orders_2026-10-02.csv?op=CREATE&user.name=etl&namenoderpcaddress=...

The second sends the data to the DataNode address from the Location header:

curl -i -X PUT -T orders_2026-10-02.csv "<Location URL from the first response>"

A 201 Created response means the file is stored. The client must be able to reach the DataNodes directly, not just the NameNode. user.name works only on clusters without Kerberos; secured clusters require SPNEGO authentication or a delegation token, and are usually reached through a gateway such as Apache Knox.

Copy between clusters and object stores with DistCp

DistCp runs a distributed job that copies files in parallel, which makes it the tool for large trees:

hadoop distcp -update -p hdfs://nn1.example.com:8020/data/raw hdfs://nn2.example.com:8020/data/raw

-update copies only files that are missing or different at the destination, so a failed run can be restarted. -p preserves attributes such as ownership and permissions. The same command works with object storage URIs, for example s3a://my-bucket/raw, once the connector and credentials are configured.

Load relational data with Spark instead of Sqoop

Sqoop imported tables over JDBC using parallel map tasks. It was retired in June 2021 and receives no fixes, including security fixes, so do not start new work with it. Spark's JDBC data source does the same job and can write straight to Parquet.

Start the Spark SQL shell with the driver for your database:

spark-sql --packages org.postgresql:postgresql:42.7.4

Then define the source table and copy it:

CREATE TEMPORARY VIEW src_orders
USING jdbc
OPTIONS (
  url 'jdbc:postgresql://db.example.com:5432/shop',
  dbtable 'public.orders',
  user 'etl_reader',
  password 'secret',
  partitionColumn 'order_id',
  lowerBound '1',
  upperBound '1000000',
  numPartitions '8',
  fetchsize '10000'
);

CREATE TABLE orders_raw
USING parquet
LOCATION 'hdfs:///data/raw/orders_parquet'
AS SELECT * FROM src_orders;

The four partitioning options are what make the read parallel. Spark issues numPartitions queries, each covering a range of partitionColumn. Things to know:

• lowerBound and upperBound set the width of each range. They do not filter rows: values outside the bounds are still read, by the first and last partitions.
• Without the partitioning options, the whole table is read over one connection by one task.
• Choose an evenly distributed, indexed numeric, date or timestamp column. A skewed column gives one huge partition and several empty ones.
• Each partition is a database connection running a range query. Eight is polite; two hundred can overload a production database. Read from a replica where you can.
• Avoid passwords on the command line and in scripts in real pipelines. Use your platform's secret store.

For continuous replication, repeated bulk reads are the wrong tool. Change data capture reads the database's transaction log and publishes every insert, update and delete as an event. Debezium is the common open-source implementation, and it delivers to Kafka.

Ingest logs and events without Flume

Flume agents collected log events and delivered them to HDFS. The Apache Logging Services project declared Flume dormant in October 2024 and advised users to migrate. Its repository has since noted a rework in progress and advises against use until a new release is announced, so treat it as unavailable for new systems.

The current pattern has two stages:

1. Collect into Apache Kafka. Applications and log shippers publish events to Kafka topics. Kafka stores them durably and lets consumers replay them.
2. Write from Kafka to storage. A Kafka Connect sink connector, a Spark Structured Streaming job or an Apache Flink job reads the topics and writes files to HDFS or object storage, typically as Parquet partitioned by date.

Separating the stages means a slow or stopped HDFS does not block the applications producing events.

Apache NiFi fits a different need: many heterogeneous sources (SFTP directories, HTTP endpoints, message queues, databases) that have to be routed, lightly transformed and delivered, with a visual flow and a record of where each piece of data came from.

Apache Storm, which older articles list next to these tools, is a stream processing engine and not an ingestion tool. It is still an active Apache project, though most new streaming work uses Flink, Spark Structured Streaming or Kafka Streams.

Common errors

Permission denied: user=alice, access=WRITE, inode="/data/raw":hdfs:supergroup:drwxr-xr-x Your user cannot write to that directory. Have an HDFS administrator create a directory you own, or grant access with hdfs dfs -chown or hdfs dfs -chmod. Do not get into the habit of running everything as the hdfs superuser.

Cannot create file ... Name node is in safe mode The NameNode is read-only, normally for a short time after startup while DataNodes report their blocks. Check with hdfs dfsadmin -safemode get and wait. If it stays on, find out why (missing DataNodes, for example) before forcing it off.

could only be written to 0 of the 1 minReplication nodes No DataNode accepted the block. They are down, full, or unreachable from the client. Run hdfs dfsadmin -report.

No such file or directory on -put The target's parent directory does not exist. Create it with hdfs dfs -mkdir -p.

The DiskSpace quota of /data/raw is exceeded The directory has a space quota. Remember that the quota counts replicated size, usually three times the file size.

Mistakes to avoid

• Many small files. Every file and block is an object in NameNode memory, and every small file becomes its own task. Batch events into files near the block size, or compact them afterwards.
• Leaving data as CSV. Land raw files as they arrive, then convert to Parquet or ORC for querying.
• Non-repeatable loads. A rerun that appends to yesterday's directory duplicates data. Write each load to its own dated path and overwrite that path on rerun.
• Reading files that are still being written. Use the staging-then-rename pattern described above.
• Unpartitioned JDBC reads of big tables. The job looks distributed and is not.

Further reading

• Apache Hadoop: File System Shell
• Apache Hadoop: WebHDFS REST API
• Apache Hadoop: DistCp Guide
• Apache Spark: JDBC To Other Databases

How to Install Cloudera Virtual Machine

In this article we will learn how to install Cloudera Distributed for Hadoop (CDH) Virtual Machine .