The Cassandra logs fill with warnings like this:
WARN Read 12 live rows and 48210 tombstone cells for query SELECT * FROM app.messages
WHERE conversation_id = ... LIMIT 50
And sometimes the read fails outright:
ERROR Scanned over 100001 tombstones during query ... query aborted (TombstoneOverwhelmingException)
The table only returns a handful of rows, so why is the read so slow? The answer is tombstones: deletions that Cassandra still has to read.
Why deletes leave tombstones
Cassandra never modifies data in place. SSTables on disk are immutable, and the same row can exist in several of them. So a delete can't erase anything. Instead it writes a marker, a tombstone, saying "this data is deleted as of time T".
A read merges every SSTable for the partition, and each tombstone it passes has to be read, kept in memory and compared to hide the older data. Tombstones are removed only when compaction runs after gc_grace_seconds has passed (default 10 days). The delay is intentional: it gives repairs time to spread the delete to every replica, so a replica that missed it doesn't bring the data back.
Where tombstones come from (often unexpectedly)
- Explicit
DELETEs, especially the queue pattern: insert work items, read the oldest, delete them. Each read now scans past everything deleted in the last 10 days. - TTLs: every value written with a TTL becomes a tombstone when it expires.
- Writing
nulls.INSERT … VALUES (…, null, …)writes a tombstone for that column. ORMs and prepared statements that bindnullfor "not set" do this on every write. Use unset values instead (supported since native protocol v4; most drivers do it when you don't bind the parameter). - Overwriting collections:
SET tags = {'a','b'}on a set, list or map writes a tombstone for the old collection before the new values. Usetags = tags + {'c'}to append. - Deleting ranges inside a wide partition leaves range tombstones that every slice query of that partition has to process.
