# Deduplication for repetitive data?

**URL:** <https://community.influxdata.com/t/deduplication-for-repetitive-data/10799>\
**Category:** Systems\
**Created:** [August 6, 2019, 5:46pm UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799 "2019-08-06T17:46:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![trevelyanuk](https://avatars.discourse-cdn.com/v4/letter/t/ba9def/32.png) [@trevelyanuk](https://community.influxdata.com/u/trevelyanuk)\
**Post date:** [August 6, 2019, 5:46pm UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799/1 "2019-08-06T17:46:22Z")

</div>

I’m monitoring some data and, although some fields continually change over time, other fields don’t (usually). In fact, they very rarely change; but I want to monitor them nevertheless for if and when they do. Data is being recorded with Telegraf and sent to InfluxDB.

Since several thousand points could have been taken with the same value, is there functionality that de-duplicates this at the InfluxDB level? Or is it possible for me to include a conditional statement when getting the input data in Telegraf? (IE get last value, only store a new point if the new value is different)?

Many thanks!

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [August 6, 2019, 8:25pm UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799/2 "2019-08-06T20:25:30Z")

</div>

Hello @trevelyanuk,

Thanks for your question. I don’t think there’s a way to handle conditional ingest with Telegraf or Kapacitor. I’m looking more into Kapacitor in case there’s something I’m missing, but right now I can suggest ingesting all of the data and setting alerts for when the data changes (assuming that you’re primarily interested in tracking the change):

> **[Create Chronograf alert rules | Chronograf 1.10 Documentation](https://docs.influxdata.com/chronograf/v1.7/guides/create-alert-rules/)**
>
> Trigger alerts by building Kapacitor alert rules in the Chronograf user interface (UI).

---

<div class="post-metadata">

**Author:** ![trevelyanuk](https://avatars.discourse-cdn.com/v4/letter/t/ba9def/32.png) [@trevelyanuk](https://community.influxdata.com/u/trevelyanuk)\
**Post date:** [August 8, 2019, 6:07pm UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799/3 "2019-08-08T18:07:31Z")

</div>

Pretty much - but also saving the SD of my Pi a bit, where this is running…

To be fair, it it only 96 points a minute extra - but, due to the infrequency of the data changing, I think I’ll write a separate script to simply check the data and then send a metric when something happens.

Thanks anyhoo!

---

<div class="post-metadata">

**Author:** ![ruairinewman](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/ruairinewman/32/2429_2.png) [@ruairinewman](https://community.influxdata.com/u/ruairinewman)\
**Post date:** [August 9, 2019, 10:53am UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799/4 "2019-08-09T10:53:16Z")

</div>

Hi,

Further to the OP’s original question: is any form of data deduplication possible with InfluxDB? We are using a v1.6.5 enterprise cluster.

Thanks

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [August 13, 2019, 7:50pm UTC](https://community.influxdata.com/t/deduplication-for-repetitive-data/10799/5 "2019-08-13T19:50:25Z")

</div>

@ruairinewman @trevelyanuk,

You could try writing a [UDF](https://docs.influxdata.com/kapacitor/v1.5/guides/socket_udf/) or just using a CL or the [API](https://docs.influxdata.com/influxdb/v1.7/tools/api/) to write your own custom scripts to either prevent or eliminate duplicates.

Also FYI, data deduplication is usually a feature provided by a database or filesystem which InfluxDB does not provide. The goal of deduplication is to reduce storage usage, but InfluxDB already has extremely efficient compression, so you don’t have to worry about data deduplication.