User defined primary key

,
1

I am handling the logs files in the kinesis stream and from Kinesis I am reading the records using Telegraf and inserting the data into Influxdb. I have defined a few tag fields as well. So when records coming from Telegraf to Influxdb all are having a unique timestamp and if I again read the same data from Kinesis to Telegraf then duplicate records are having a unique timestamp as well and those duplicate records get inserted in Influxdb.
So, Now what is the method to handle duplicate records so that they don’t go in influxdb.
Is there any way to define a user-defined primary key?

2

The equivalent of a “Primary Key” in InfluxDB is composed of:

  • Series
    • Measurement
    • Tag Set (keys + values)
    • Field Key
  • Timestamp

Points will be updated based on the above-defined key.
You can’t define any other “key”

I’m no expert in Kinesis, but this is what I’ve found, maybe it will help.

In the request, you can specify the shard iterator type AT_TIMESTAMP to read records from an arbitrary point in time, TRIM_HORIZON to cause ShardIterator to point to the last untrimmed record in the shard in the system (the oldest data record in the shard), or LATEST so that you always read the most recent data in the shard.

The Telegraf input supports both

  ## Shard iterator type (only 'TRIM_HORIZON' and 'LATEST' currently supported)
  # shard_iterator_type = "TRIM_HORIZON"