Busk: Created page with "In Loki, deduplication happens both at ingest-time and (optionally) at query-time, but it’s driven by the combination of: 1. Stream identity (tenant + label set, e.g. `..."

2025-07-05T16:29:22Z

Created page with "In Loki, deduplication happens both at ingest-time and (optionally) at query-time, but it’s driven by the combination of: 1. **Stream identity** (tenant + label set, e.g. `..."

New page

In Loki, deduplication happens both at ingest-time and (optionally) at query-time, but it’s driven by the combination of:

1. **Stream identity** (tenant + label set, e.g. `{job="myjob"}`),
2. **Timestamp** (the epoch you push), and
3. **Log line content**

---

**1. Ingest-time dropping of exact duplicates**
When you push two *identical* log lines (same timestamp, same labels, same exact text) into the *same* stream, the second one is silently dropped. Loki’s distributor dedupes logs that would collide on “timestamp + stream”; only the first one survives ([community.grafana.com][1], [community.grafana.com][2]).

---

**2. `increment_duplicate_timestamp` for near-duplicates**
If you enable in your `loki.yaml` under `limits_config`:

```yaml
limits_config:
increment_duplicate_timestamp: true
```

then on ingest, whenever a new line arrives with the *same timestamp* **but different content**, Loki will nudge the timestamp forward by one nanosecond to preserve order. This only applies when the log *text differs*—truly identical lines still collide and get deduped as above ([community.grafana.com][1], [community.grafana.com][1]).

---

**3. Query-time deduplication in HA setups**
In a multi-replica deployment (i.e. you’ve set a replication factor > 1), each ingester may hold its own copy of a stream. When you run a query, the Querier automatically deduplicates across ingesters so you don’t see each replica’s copy twice ([community.grafana.com][2], [saikiranpikili.medium.com][3]).

---

### So:

* **Two identical messages** (same epoch, same `job` label, same text) → only one stored.
* **Two messages with same epoch and `job` but *different* text** → both can be stored (and ordering preserved if you’ve enabled `increment_duplicate_timestamp`).
* **Multiple replicas** → query-time dedupe prevents you from seeing the same event again.

[1]: https://community.grafana.com/t/loki-drops-duplicate-log-entries/144464?utm_source=chatgpt.com "Loki drops duplicate log entries - Grafana Labs Community Forums"
[2]: https://community.grafana.com/t/does-loki-deduplicates-messages/124945 "Does Loki deduplicates messages? - Grafana Loki - Grafana Labs Community Forums"
[3]: https://saikiranpikili.medium.com/grafana-loki-all-you-need-to-know-91682e0708ce?utm_source=chatgpt.com "Grafana Loki: all you need to know | by Sai Kiran Pikili - Medium"

Loki duplicates - Revision history

Busk: Created page with "In Loki, deduplication happens both at ingest-time and (optionally) at query-time, but it’s driven by the combination of: 1. **Stream identity** (tenant + label set, e.g. `..."

Busk: Created page with "In Loki, deduplication happens both at ingest-time and (optionally) at query-time, but it’s driven by the combination of: 1. Stream identity (tenant + label set, e.g. `..."