InfluxDB was fast for months. Then writes started lagging, queries timed out, and the process kept getting OOM-killed. Nothing changed except "we added some tags". Welcome to series cardinality, the most common way to break an InfluxDB 1.x or 2.x deployment.
What a series is
InfluxDB line protocol has four parts:
cpu,host=web-01,region=eu-west usage_user=12.5,usage_system=3.1 1790000000000000000
│ └──────── tags ─────────┘ └──────────── fields ───────────┘ └── timestamp ──┘
measurement
- Tags are indexed metadata, stored as strings.
- Fields are the actual values, and aren't indexed.
A series is each unique combination of measurement + tag set (per field). The storage engine (TSM plus the TSI index) keeps a structure per series. Series cardinality is how many distinct series exist: the product of the distinct values of every tag.
3 regions × 200 hosts × 4 metrics = 2,400 series ✓
... × request_id (a new value per request) = unbounded ✗
How it breaks
With unbounded tag values (user IDs, request or trace IDs, session IDs, container or pod IDs that change on every deploy, full URLs, error messages), cardinality grows without limit:
- the series index outgrows memory and gets slow to update, so writes slow down;
- queries that match many series scan huge index ranges, so queries time out;
- memory use keeps climbing until the process is OOM-killed, and startup slows as the index loads;
- on InfluxDB Cloud (TSM-based plans), you hit cardinality limits and writes are rejected.
Measure it
InfluxDB 1.x (InfluxQL):
SHOW SERIES CARDINALITY ON telegraf;
SHOW TAG VALUES CARDINALITY ON telegraf FROM cpu WITH KEY = "host";
InfluxDB 2.x (Flux):
import "influxdata/influxdb"
influxdb.cardinality(bucket: "telegraf", start: -30d)
import "influxdata/influxdb/schema"
schema.tagValues(bucket: "telegraf", tag: "request_id") |> count()
Find the tag with the most distinct values. That's nearly always the culprit.
Fix it
1. Move high-cardinality values from tags to fields. If you never GROUP BY or filter on the value by exact match, it doesn't need to be a tag:
# before: new series per request
http,path=/api/orders,request_id=8f3c… latency=42
# after: request_id is data, not index
http,path=/api/orders latency=42,request_id="8f3c…"
2. Bound the values you keep as tags. Normalize URLs to route templates (/orders/:id, not /orders/81723), bucket status codes (2xx, 5xx), and map ephemeral pod names to a stable service or deployment tag.
3. Drop tags nobody queries. Telegraf and other agents add many by default. Use tagexclude / taginclude or processors to remove them before writing.
4. Expire old series. Set a retention period on the bucket and downsample old data into a separate bucket with fewer tags. Series that stop receiving data disappear once their data ages out.
5. Clean up what's already there. Deleting data doesn't always drop series from the index immediately. In 1.x, DROP SERIES FROM … WHERE …; in 2.x, influx delete with a predicate. After a big cleanup, rebuilding the TSI index (influx_inspect buildtsi in 1.x, offline) reclaims memory.
What about InfluxDB 3?
InfluxDB 3 (Core, Enterprise and the current Cloud products) is rebuilt on a columnar engine (Apache Arrow, DataFusion, Parquet) and no longer keeps a per-series in-memory index. It's designed to handle far higher cardinality, which makes things like per-request tracing data practical.
Schema discipline still pays off there: fewer, well-chosen tag columns mean smaller files, better compression and faster queries. And upgrading from 1.x or 2.x to 3 is a data migration, not an in-place upgrade.
Checklist
- Series = measurement + tag set. Every distinct tag value multiplies the series count.
- Unbounded tags (IDs, URLs, pod names) cause slow writes, timeouts and OOMs on 1.x/2.x.
- Measure with
SHOW SERIES CARDINALITYorinfluxdb.cardinality(). - Move IDs to fields, normalize tag values, drop unused tags, and set retention.
Get the weekly commit
New database deep dives every week.
