Big data is data that a single machine running a conventional database cannot store or process fast enough, so the work has to be split across many machines. It is a statement about the gap between your data and your tools, not a fixed number of gigabytes. A dataset that needed a cluster ten years ago may fit comfortably on one server today.
That definition matters because the tools built for big data are more complex and more restrictive than a normal database. You want them only when you have the problem they solve.
Where the term came from
In the early 2000s Google had more web pages to index than any one computer could handle. It published two papers describing its answer: the Google File System (2003), which stored files across thousands of cheap machines, and MapReduce (2004), a way of writing a computation so that it could run on all of them in parallel and survive individual machines failing.
Engineers working on an open-source search engine reimplemented those ideas as Apache Hadoop, first released in 2006. For the first time, any company could process very large datasets on commodity hardware, and "big data" became the name for both the datasets and the family of tools that grew around Hadoop.
The characteristics: volume, velocity, variety
The usual description, first proposed by analyst Doug Laney in 2001, is three Vs:
| Characteristic | What it means | Example |
|---|---|---|
| Volume | More data than one machine can store or scan in reasonable time | Years of clickstream events from a large website |
| Velocity | Data arrives continuously and is useful only if processed quickly | Payment events checked for fraud before approval |
| Variety | Data comes in many shapes, not just rows and columns | Database tables, JSON logs, images, free text |
Two more are often added. Veracity is how far you can trust the data: sensor glitches, duplicates and missing fields all grow with scale. Value is the reminder that collecting data is a cost until someone uses it to make a decision.
Any one of the first three can push you beyond a conventional database. A modest volume arriving as a million events per second is a big data problem. So is a modest volume made of video files.
Structured, semi-structured and unstructured data
Variety is usually described in three groups:
- Structured data has a fixed schema: rows in a relational table.
- Semi-structured data describes itself but varies from record to record: JSON, XML, log lines.
- Unstructured data has no predefined model: text documents, audio, images, video.
Relational databases were designed for the first group. Much of the big data toolset exists to handle the other two.
Do you actually have big data?
Many teams that assume they do, do not. Work through these questions before adopting a distributed system:
- Does the data you actually query fit on one machine? Count the working set, not the archive. You might keep ten years of history and query the last ninety days.
- Does a query finish in acceptable time on one machine? A columnar engine scanning compressed files on a single large server can handle far more than a row-oriented database on the same hardware.
- Can one machine keep up with the write rate? Measure it before assuming not: a tuned relational database on modern storage sustains a high rate of small writes.
- Is the bottleneck the data, or the query? A missing index or a row-oriented database doing analytical scans looks like a scale problem and is not one.
If every answer is yes, you have ordinary data, and a relational database for transactions plus a columnar engine for analysis will be simpler, cheaper and faster than a cluster. For example, DuckDB, an analytical database that runs inside a single process, queries a directory of Parquet files directly:
SELECT
date_trunc('day', event_time) AS day,
count(*) AS events,
count(DISTINCT user_id) AS users
FROM read_parquet('events/*.parquet')
GROUP BY day
ORDER BY day;
No cluster, no loading step and no server to manage. If a query like that over your data takes seconds on a laptop or one cloud instance, a distributed platform will add cost without adding speed.
You do have big data when the dataset is larger than the storage you can attach to one machine, when single-node queries take hours no matter how you tune them, when events arrive faster than one node can ingest, or when the data must stay available through the loss of a whole machine or data centre.
How big data systems work
Every big data system rests on the same three ideas.
Partitioning. The data is split into chunks, by key or by time range, and the chunks are spread over many machines. A query that needs only some chunks reads only those.
Replication. Each chunk is stored on several machines. With hundreds of machines, hardware failure is routine, and the system has to keep working through it.
Moving the computation to the data. Instead of pulling everything to one place, each machine processes its own chunks and sends back a small result, which a coordinator combines. MapReduce was the first widely used form of this. Current engines do the same thing with far less code and far more speed.
These ideas have a price. Joins across partitions are expensive. Transactions spanning many machines are slow or unavailable. Results may be eventually consistent. This is the trade you make to get scale, and the reason not to make it early.
Batch and stream processing
Batch processing runs over a bounded dataset that already exists: last night's sales, last month's logs. It favours throughput and is the simpler model.
Stream processing runs continuously over events as they arrive and produces results within seconds or less. It suits fraud detection, monitoring and live dashboards, and it is harder to get right because events arrive late, out of order and sometimes twice.
Start with batch unless the business case depends on low latency. Many "real-time" requirements turn out to be satisfied by a batch job that runs every few minutes.
The current tool categories
The original Hadoop stack bundled storage, processing and resource management into one cluster you ran yourself. Current platforms separate those layers, and most of them run on cloud object storage.
| Layer | What it does | Examples |
|---|---|---|
| Object storage | Cheap, durable storage for files of any type | Amazon S3, Google Cloud Storage, Azure Blob Storage |
| File formats | Compressed, columnar files for analytics | Apache Parquet, Apache ORC |
| Table formats | Make a set of files behave like a table, with schema changes, snapshots and atomic commits | Apache Iceberg, Delta Lake, Apache Hudi |
| Distributed processing | Run transformations and SQL over many machines | Apache Spark, Trino, Apache Flink |
| Cloud data warehouses | Managed SQL analytics with storage and compute scaled separately | Snowflake, Google BigQuery, Amazon Redshift, Databricks SQL |
| Event streaming | Durable, ordered logs of events between systems | Apache Kafka |
| Real-time analytics | Fast aggregations over fresh data | ClickHouse, Apache Druid, Apache Pinot |
| Distributed operational databases | Transactional or key-based workloads beyond one node | Apache Cassandra, CockroachDB, Google Spanner |
| Orchestration | Schedule and monitor pipelines | Apache Airflow |
You will meet two architecture names. A data warehouse stores cleaned, structured data for SQL analysis. A data lake stores raw files of any kind in object storage. A lakehouse puts a table format over the lake so that the same files can be queried with warehouse-style SQL and guarantees.
Common mistakes
- Buying the tools before having the problem. A cluster for a dataset that fits in memory is slower than one machine, because coordination has a cost.
- Collecting everything and deciding later. Storage is cheap, but unlabelled, unowned data becomes a "data swamp" nobody can use, and personal data you hold is a legal obligation under laws such as GDPR whether you use it or not.
- Ignoring data quality. At scale you cannot inspect rows by eye. Validate schemas and value ranges at the point of ingestion.
- Many tiny files. Distributed engines are efficient with fewer, larger files. Thousands of kilobyte-sized files make every query slow.
- Assuming more data means better answers. A biased or wrong dataset stays biased or wrong at any size.
Further reading
Get the weekly commit
New database deep dives every week.
