You migrated a busy MySQL table to TiDB for horizontal scale. You added TiKV nodes, and write throughput barely moved. The TiDB Dashboard shows one TiKV node at 100% CPU while the others are idle.
It's almost always the same cause: an auto-increment primary key creating a write hotspot.
How TiDB stores rows
TiDB stores table data in TiKV as sorted key-value pairs, split into Regions of about 96 MB. Each Region is a contiguous key range served by one Raft leader on one TiKV node. Horizontal scaling works by spreading Regions, and so the load, across nodes.
With a clustered integer primary key (the default for single-column integer PKs in recent versions), the row key is the primary key. With AUTO_INCREMENT, every new row gets the next number, so every insert goes to the end of the key space, which is one Region on one node. When that Region splits, the new tail Region takes over as the hotspot. Adding nodes doesn't help, because all writes still land in one place.
Tables without a primary key, or with a NONCLUSTERED one, have the same problem with the hidden _tidb_rowid, which also increases monotonically.
Confirm it
- TiDB Dashboard → Key Visualizer: a bright diagonal line along the top of a table's key range is the signature of sequential writes.
- Grafana → TiKV-Details: one store with much higher write traffic or CPU than the rest.
information_schema.TIDB_HOT_REGIONS(orTIDB_HOT_REGIONS_HISTORY) lists the hottest Regions and their tables.
Fix: AUTO_RANDOM for clustered integer keys
Replace AUTO_INCREMENT with AUTO_RANDOM:
CREATE TABLE orders (
id BIGINT PRIMARY KEY AUTO_RANDOM,
customer_id BIGINT NOT NULL,
created_at DATETIME NOT NULL,
total DECIMAL(, )
);
