Dedup processor plugin

1

Hi everyone, I have recently used the influx system.
I use telegraf to insert points in influxdb.
I wanted to use dedup plugin to filter and not insert into the db
metrics that are the exact repetition, unless the timestamp, of the previous value.
Could I have an example of using the dedup process plugin?

telegraf/README.md at master · influxdata/telegraf · GitHub

Thanks in advance
sp

2

Hello @salpas,
What do you mean by

I wanted to use dedup plugin to filter and not insert into the db
metrics that are the exact repetition, unless the timestamp, of the previous value.

I found this a little confusing. Can you please elaborate or provide an example of what you mean?

3

Hi Anaisdg, thanks for the reply.

I try to explain the matter better.
I use telegraf to insert points in influxdb.
I want to use dedup plugin so that, if the next point is the same as the previous one, it is not inserted in the db

Es.:
Arrived point :
cpu,cpu=cpu0 time_idle=44i,time_guest=2i
telegram insert in db influx
next poit:
cpu,cpu=cpu0 time_idle=44i,time_guest=2i
it is the same of previus, telegraf discarded this point and not insert in db.

is it possible using dedup plugin process?

if Yes, is possible un examples?

thanks
sp

4

The docs on GitHub are not the best on this one, I had a quick look at it and its only parameter should be the cache duration for values.

  1. Passing values are cached
  2. upcoming data are matched by key (measurement, tags key and values, field keys)
  3. if any field value is different from the cached ones, the data is written to the output, and saved in the cache (replacing the previus one)
  4. if all the field values are equals to the cached ones, the cache “lifetime” is refreshed and nothing is sent to the output

ofc for a key there will be only one cached “row”, the latest

As any other processor, it works on all the data unless you filter it (see filtering)

The usage should be as simple as putting this in the conf file.

[[processors.dedup]]
  ## Maximum time to suppress output
  dedup_interval = "600s"

just adjust the cache duration to your gathering interval, the lower the frequency, the higher the cache duration

5

Hi giovanni, thanks for your replay.
I tried to use it by putting it in the telegraf.conf file, but it doesn’t seem to work.
What am I doing wrong?
Below is my configuration file.
Thanks in advance for any further feedback

# Telegraf Configuration

# Global tags can be specified here in key=“value” format.
[global_tags]

# Configuration for telegraf agent
[agent]

  • interval = “1s”*
  • round_interval = true*
  • metric_batch_size = 20*
  • metric_buffer_limit = 10000*
  • collection_jitter = “0s”*
  • flush_interval = “1s”*
  • flush_jitter = “0s”*
  • precision = “”*
  • debug = true*
  • logfile = “/var/log/telegraf/telegraf.log”*
  • hostname = “”*
  • omit_hostname = false*

###############################################################################
# OUTPUTS #
###############################################################################

# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]

  • urls = [“http://127.0.0.1:8086”]*

  • database = “BMS_LOGGER_RAW”*

  • skip_database_creation = false*

  • retention_policy = “”*

###############################################################################
# PROCESSORS #
###############################################################################
[[processors.dedup]]

  • dedup_interval = “600s”*

###############################################################################
# INPUTS #
###############################################################################

# Windows Performance Counters plugin.
# These are the recommended method of monitoring system metrics on windows,
# as the regular system plugins (inputs.cpu, inputs.mem, etc.) rely on WMI,
# which utilize more system resources.
#
# See more configuration examples at:
# telegraf/plugins/inputs/win_perf_counters at master · influxdata/telegraf · GitHub

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/120.txt”]*

  • name_override=“120_general_1”*

  • tagexclude = [“path”,“host”]*

  • from_beginning = false*

  • watch_method = “poll” *

  • pipe = false*

  • data_format = “grok”*

  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:batteryCurrent120}%{WORD2:batteryVoltage120}%{WORD3:stateOfCharge}%{WORD4:stateOfHealth}%{WORD5:bmsState}%{WORD6:aliveCounter}’]*

  • grok_custom_patterns = ‘’'*

  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})	*

  •   WORD5 (?:[0-9A-F]{2})	*

  •   WORD6 (?:[0-9A-F]{2})	*

  • ‘’'*

[[processors.converter]]

  • order = 1*
  • [processors.converter.fields]*
  • unsigned = [“stateOfCharge”,“stateOfHealth”,“aliveCounter”]*
  • integer = [“batteryCurrent120”]*
  • float = [“batteryVoltage120”]*

[[processors.compute]]

  • order = 2*
  • missing = “ignore”*
  • [processors.compute.fields]*
  •   batteryCurrent120 = "batteryCurrent120 / 16.0"*

  •   batteryVoltage120 = "batteryVoltage120 / 16.0"*

  •   *

[[inputs.tail]]

  • Log files to parse.*

  • files = [“/home/root/SlowWriteCanMW/121.txt”]*
  • name_override=“121_general_2”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:remainingCapacity121}%{WORD2:remainingEnergy}%{WORD3:chargeState}%{WORD4:reserved}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*
  • ‘’'*

[[processors.converter]]

  • order = 3*
  • [processors.converter.fields]*
  • unsigned = [“remainingEnergy”]*
  • float = [“remainingCapacity121”] *
    [[processors.compute]]
  • order = 4*
  • missing = “ignore”*
  • [processors.compute.fields]*
  •   remainingCapacity121 = "remainingCapacity121 / 16.0"*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/122.txt”]*
  • name_override=“122_general_fast”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:batteryCurrent122}%{WORD2:highestCellVoltage}%{WORD3:lowestCellVoltage}%{WORD4:contactorOpeningPending}%{WORD5:lowestCellNumber}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{4})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})*
  • ‘’'*

[[processors.converter]]

  • order = 5*
  • [processors.converter.fields]*
  • unsigned = [“highestCellVoltage”,“lowestCellVoltage”,“lowestCellNumber”]*
  • integer = [“batteryCurrent122”]*

[[processors.compute]]

  • order = 6*
  • missing = “ignore”*
  • [processors.compute.fields]*
  •   batteryCurrent122 = "batteryCurrent122 / 16.0"*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/123.txt”]*
  • name_override=“123_voltages”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:lowestCellVoltage123}%{WORD2:lowestCellVoltageNumber123}%{WORD3:highestCellVoltage123}%{WORD4:highestCellVoltageNumber123}%{WORD5:averageCellVoltage123}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{4})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{4})*
  • ‘’'*

[[processors.converter]]

  • order = 7 *
  • [processors.converter.fields]*
  • unsigned = [“lowestCellVoltage123”,“lowestCellVoltageNumber123”,“highestCellVoltage123”,“highestCellVoltageNumber123”,“averageCellVoltage123”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/124.txt”]*

  • name_override=“124_temperatures”*

  • tagexclude = [“path”,“host”]*

  • from_beginning = false*

  • watch_method = “poll” *

  • pipe = false*

  • data_format = “grok”*

  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:lowestTemperatureSensorValue124}%{WORD2:lowestTemperatureSensorNumber124}%{WORD3:highestTemperatureSensorValue124}%{WORD4:highestTemperatureSensorNumber124}%{WORD5:averageCellsTemperature124}%{WORD6:boardLogicTemperature124}%{WORD7:reserved}%{WORD8:maxBalancingHeatsinkTemperature124}’]*

  • grok_custom_patterns = ‘’'*

  • WORD1 (?:[0-9A-F]{2})*

  • WORD2 (?:[0-9A-F]{2})*

  • WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})*

  •   WORD6 (?:[0-9A-F]{2})*

  •   WORD7 (?:[0-9A-F]{2})*

  •   WORD8 (?:[0-9A-F]{2})		*

  • ‘’'*
    [[processors.converter]]

  • order = 8*

  • [processors.converter.fields]*

  • unsigned = [“lowestTemperatureSensorNumber124”,“highestTemperatureSensorNumber124”]*

  • integer = [“lowestTemperatureSensorValue124”,“highestTemperatureSensorValue124”,“averageCellsTemperature124”,“boardLogicTemperature124”,“maxBalancingHeatsinkTemperature124”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/125.txt”]*
  • name_override=“125_resistance”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:lowestCellResistance125}%{WORD2:lowestCellResistanceNr125}%{WORD3:highestCellResistance125}%{WORD4:highestCellResistanceNr125}%{WORD5:averageCellResistance125}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{2})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})	*
  • ‘’'*

[[processors.converter]]

  • order = 9*
  • [processors.converter.fields]*
  • unsigned = [“lowestCellResistanceNr125”,“highestCellResistanceNr125”]*
  • float = ["lowestCellResistance125","highestCellResistance125","averageCellResistance125"]*

[[processors.compute]]

  • order = 10*
  • missing = “ignore”*
  • [processors.compute.fields]*
  •   lowestCellResistance125 = "lowestCellResistance125 / 8.0"*

  •   highestCellResistance125 = "highestCellResistance125 / 8.0"*

  •   averageCellResistance125 = "averageCellResistance125 / 8.0"*

  •   *

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/126.txt”]*
  • name_override=“126_setpoints”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:chargeCurrentSetpoint126}%{WORD2:chargeVoltageSetpoint126}%{WORD3:dischargeCurrentSetpoint126}%{WORD4:maxRegCurrentSetpoint126}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{4})*

  •   WORD4 (?:[0-9A-F]{4})*
  • ‘’'*

[[processors.converter]]

  • order = 11*
  • [processors.converter.fields]*
  • unsigned = [“chargeVoltageSetpoint126”]*
  • float = [“chargeCurrentSetpoint126”,“dischargeCurrentSetpoint126”,“maxRegCurrentSetpoint126”]*
    [[processors.compute]]
  • order = 12*
  • missing = “ignore”*
  • [processors.compute.fields]*
  •   chargeCurrentSetpoint126 = "chargeCurrentSetpoint126 / 16.0"*

  •   dischargeCurrentSetpoint126 = "dischargeCurrentSetpoint126 / 16.0"*

  •   maxRegCurrentSetpoint126 = "maxRegCurrentSetpoint126 / 16.0"*

  •   *

  •   *

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/130.txt”]*
  • name_override=“130_errors”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:warnings130}%{WORD2:errors130}%{WORD3:faults130}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{4})*
  • ‘’'*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/131.txt”]*

  • name_override=“131_values”*

  • tagexclude = [“path”,“host”]*

  • from_beginning = false*

  • watch_method = “poll” *

  • pipe = false*

  • data_format = “grok”*

  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:cellVoltage131}%{WORD2:cellNumber131}%{WORD3:tempValue131}%{WORD4:tempNumber131}’]*

  • grok_custom_patterns = ‘’'*

  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})		*

  • ‘’'*

[[processors.converter]]

  • order = 13*
  • [processors.converter.fields]*
  • unsigned = [“cellVoltage131”,“cellNumber131”,“tempNumber131”]*
  • integer = [“tempValue131”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/132.txt”]*

  • name_override=“132_io”*

  • tagexclude = [“path”,“host”]*

  • from_beginning = false*

  • watch_method = “poll” *

  • pipe = false*

  • data_format = “grok”*

  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:ioStatusByte0_132}%{WORD2:ioStatusByte1_132}%{WORD3:ioStatusByte2_132}%{WORD4:ioStatusByte3_132}%{WORD5:ioStatusByte4_132}’]*

  • grok_custom_patterns = ‘’'*

  •   WORD1 (?:[0-9A-F]{2})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})			*

  • ‘’'*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/133.txt”]*
  • name_override=“133_generics_1”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:reserved1_133}%{WORD2:reserved2_133}%{WORD3:reserved3_133}%{WORD4:reserved4_133}%{WORD5:energyIn133}%{WORD6:energyOut133}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{2})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{4})*

  •   WORD6 (?:[0-9A-F]{4})*
  • ‘’'*

[[processors.converter]]

  • order = 14*
  • [processors.converter.fields]*
  • unsigned = [“energyIn133”,“energyOut133”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/134.txt”]*
  • name_override=“134_ui”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:commandFromDashboard}%{WORD2:reserved2_134}%{WORD3:reserved3_134}%{WORD4:reserved4_134}%{WORD5:reserved5_134}%{WORD6:reserved6_134}%{WORD7:energyIn134}%{WORD8:reserved7_134}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{2})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})*

  •   WORD6 (?:[0-9A-F]{2})*

  •   WORD7 (?:[0-9A-F]{2})*

  •   WORD8 (?:[0-9A-F]{2})		*
  • ‘’'*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/135.txt”]*
  • name_override=“135_generic_2”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:capacityIn135}%{WORD2:capacityOut135}%{WORD3:loadCycles135}%{WORD4:uvFaultsCounter135}%{WORD5:otFaultsCounter135}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{4})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{2})*
  • ‘’'*

[[processors.converter]]

  • order = 15*
  • [processors.converter.fields]*
  • unsigned = [“capacityIn135”,“capacityOut135”,“loadCycles135”,“uvFaultsCounter135”,“otFaultsCounter135”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/136.txt”]*
  • name_override=“136_generic_3”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:uvErrorCounter136}%{WORD2:otErrorCounter136}%{WORD3:uvWarningCounter136}%{WORD4:otWarningCounter136}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{4})*

  •   WORD2 (?:[0-9A-F]{4})*

  •   WORD3 (?:[0-9A-F]{4})*

  •   WORD4 (?:[0-9A-F]{4})*
  • ‘’'*

[[processors.converter]]

  • order = 16*
  • [processors.converter.fields]*
  • unsigned = [“uvErrorCounter136”,“otErrorCounter136”,“uvWarningCounter136”,“otWarningCounter136”]*

[[inputs.tail]]

  • files = [“/home/root/SlowWriteCanMW/137.txt”]*
  • name_override=“137_details_bcu”*
  • tagexclude = [“path”,“host”]*
  • from_beginning = false*
  • watch_method = “poll” *
  • pipe = false*
  • data_format = “grok”*
  • grok_patterns = [‘%{NUMBER:timestamp:ts-epoch},%{WORD1:fwVersionMajor}%{WORD2:fwVersionMinor}%{WORD3:fwVersionPatch}%{WORD4:bcuVers}%{WORD5:bcuSn}’]*
  • grok_custom_patterns = ‘’'*
  •   WORD1 (?:[0-9A-F]{2})*

  •   WORD2 (?:[0-9A-F]{2})*

  •   WORD3 (?:[0-9A-F]{2})*

  •   WORD4 (?:[0-9A-F]{2})*

  •   WORD5 (?:[0-9A-F]{8})		*
  • ‘’'*
6

tbh, I’m not sure…

I’d start by adding the order = nn for the dedup processor too, try that and put it as the last one.
as of now I’m not sure about when is that being executed… I hope after all the other but I cannot guarantee that.

it would be helpful to have a proper formatted conf file, put it in a codeblock next time (wrap it in “```”)