Big Data: The Future Has Arrived

The vendors consolidated at the same time. Cloudera and Hortonworks merged in a deal that closed in January 2019. MapR's assets were bought by Hewlett Packard Enterprise later that year. Cloudera was taken private in 2021 and now positions its platform around hybrid cloud deployment and Iceberg tables.

Hadoop is not gone: the Apache project still publishes releases, and many large organisations run clusters that work and are paid for. It is simply no longer where new platforms start.

Why the architecture changed

Three shifts undid the assumptions Hadoop was built on.

Storage and compute came apart. In a Hadoop cluster, adding storage meant adding servers, CPUs included, and the cluster ran day and night to keep the data available. Fast networks made data locality unnecessary, so data could sit in cheap object storage while compute clusters were started for a job and shut down afterwards. You pay for storage by the gigabyte and for compute only while queries run. Object stores also became easier to build on: Amazon S3 has given strong read-after-write consistency since December 2020, removing a whole class of workarounds.

Columnar files plus table formats replaced "a directory of files". A Hive table was a folder; a query listed the folder and read what was there. That gave no safe concurrent writes, no way to update or delete a row without rewriting partitions, and slow planning on large tables. Open table formats keep data in Parquet files and add a transaction log of metadata that records exactly which files make up each version of the table. The result is ACID commits, row-level UPDATE, DELETE and MERGE, schema changes without rewriting data, and the ability to query the table as it was at an earlier time.

SQL won. Writing MapReduce jobs in Java turned out to be a bottleneck, not a feature. Cloud warehouses (Snowflake, Google BigQuery, Amazon Redshift) showed that analysts would rather write SQL against a service that needs no cluster administration, and the open-source engines followed.

The lakehouse

A data lake was cheap, flexible file storage with weak guarantees. A data warehouse was a managed SQL database with strong guarantees and a proprietary storage format. The lakehouse is the name for combining them: warehouse behaviour (transactions, schemas, fast SQL) on top of open files in object storage that several engines can read.

Creating and changing an Iceberg table from Spark SQL looks like ordinary database work:

CREATE TABLE lake.web.page_views (
    view_id     bigint,
    user_id     bigint,
    url         string,
    viewed_at   timestamp
)
USING iceberg
PARTITIONED BY (days(viewed_at));

INSERT INTO lake.web.page_views VALUES
    (1, 1001, '/pricing', TIMESTAMP '2026-10-01 09:15:00'),
    (2, 1002, '/docs',    TIMESTAMP '2026-10-01 09:16:30');

DELETE FROM lake.web.page_views WHERE user_id = 1002;

SELECT snapshot_id, committed_at, operation
FROM lake.web.page_views.snapshots
ORDER BY committed_at;

The DELETE is a real row-level delete, which matters for privacy regulations that require erasing one person's data. The last query lists the table's snapshots, one for the insert and one for the delete. Take the snapshot_id of the insert and you can read the table as it was before the delete:

SELECT count(*) FROM lake.web.page_views VERSION AS OF 4348509562937412345;

Replace the number with a snapshot_id from your own output; TIMESTAMP AS OF '2026-10-08 12:00:00' works the same way with a point in time. Here lake is whatever name you gave the Iceberg catalogue in your Spark configuration, and row-level DELETE needs the Iceberg SQL extensions enabled in that session. days(viewed_at) is hidden partitioning: queries filter on viewed_at directly and the engine prunes files, with no separate partition column to remember.

The same files can then be read by another engine. With the catalogue attached, Trino, Snowflake, BigQuery, Athena and DuckDB can all query Iceberg tables. That interoperability is why Iceberg has become the format the large vendors converge on, including Databricks, which created Delta Lake and bought the company founded by Iceberg's creators in 2024.

Not all data is big

The other correction since 2016 is about size. Hardware grew faster than most datasets. A single cloud server can have hundreds of gigabytes of memory, and columnar engines such as DuckDB and ClickHouse process billions of rows on one machine. A great deal of work that was pushed onto clusters a decade ago now fits comfortably on one node.

A query over Parquet files in object storage, from a laptop, with DuckDB:

INSTALL httpfs;
LOAD httpfs;

SELECT date_trunc('day', viewed_at) AS day, count(*) AS views
FROM read_parquet('s3://example-bucket/page_views/*.parquet')
GROUP BY day
ORDER BY day;

Replace the bucket path with your own. For a private bucket, give DuckDB credentials first with a CREATE SECRET statement of type s3.

Before designing a distributed system, measure the data. If the tables you actually query are tens of gigabytes, a single PostgreSQL server, a small warehouse or DuckDB will be simpler, cheaper and faster than a cluster.

If you still run Hadoop

A migration usually goes in this order:

1. Inventory by usage, not by size. Much of what sits in HDFS is never read. Find the tables and jobs that are actually used.
2. Move storage first. Copy data to object storage (hadoop distcp writes to S3-compatible stores through the s3a:// connector) and point existing Hive tables at the new location.
3. Convert table formats. Iceberg provides procedures to register or migrate existing Parquet and ORC Hive tables without rewriting the data files.
4. Move processing. Spark jobs carry over with configuration changes. Hive SQL needs testing on the new engine; dialect differences are small but real. Pig scripts and MapReduce jobs have to be rewritten.
5. Replace the periphery. Oozie workflows become Airflow DAGs. Sqoop imports become CDC pipelines or scheduled extracts.
6. Rebuild security deliberately. Kerberos and Ranger policies do not translate automatically to cloud IAM and catalogue permissions.

Common mistakes in the new stack

• Small files. Streaming or frequent small inserts create thousands of tiny Parquet files and slow every query. Schedule compaction; every table format provides it.
• Never expiring snapshots. Time travel keeps old data files. Without snapshot expiry, storage grows indefinitely, and "deleted" personal data is still physically present.
• Forgetting that elastic means unbounded. A warehouse that scales automatically will also bill automatically. Set quotas and query timeouts on day one.
• Treating object storage like a file system. Listing is slow and renames are copies. Let the table format track files; do not build logic on directory listings.
• Over-partitioning. Partitioning by a high-cardinality column produces the small-files problem by design. Partition by day or month at most, and let the per-file column statistics do the rest.

Further reading

• Apache Iceberg documentation
• Delta Lake documentation
• Apache Hadoop releases
• Apache Attic: retired projects

How do I insert a row only if it doesn't exist, without race conditions?

Select-then-insert breaks under concurrency. Use a unique constraint plus one atomic statement: ON CONFLICT, ON DUPLICATE KEY UPDATE, or UPDLOCK/SERIALIZABLE in SQL Server.

#concurrency#schema-design#transactions